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ABSTRACT 



This report describes the second-year results for an 
evaluation of the School Choice Scholarships Foundation (SCSF) program to 
award 1,300 scholarships so that children of low-income families in grades 1 
through 4 in New York City public schools could transfer to private schools. 
Because the scholarships were awarded through a lottery, the evaluation 
allowed for the conduct of a randomized experiment in which students were 
randomly selected for a treatment (scholarship) group and a control group. As 
reported by parents, the schools attended by scholarship students were 
smaller than the schools attended by public school students. Private school 
parents were less likely to report serious problems such as fighting, 
absenteeism, and racial conflict at their children's schools. Students in 
private schools were asked to complete more homework than students in public 
schools. On standardized tests, students offered scholarships generally 
performed at about the same level as students in the control group. African 
American students offered scholarships scored about three points higher than 
similar students in the control group, and the impact of going to a private 
school for two years for African American students was four percentile 
points. No impact was seen for Latino students. Overall, findings suggested 
that the difference between the scholarship group and the control group 
remained about the same over the two years. About 62 percent of the students 
offered scholarships used them for two full years, and 12 percent used them 
in the first year but not the second. 24 percent of students offered the 
scholarships refused them. The most frequently cited obstacles that prevented 
parents from sending their children to the preferred school included cost (35 
percent) , transportation problems (14 percent) , and lack of space at the 
school (10 percent) . Findings show that 40 percent of parents who switched 
from public to private schools gave their schools an A; less than 10 percent 
of similar parents in the public schools gave their schools an A. Five 
appendixes contain discussions of baseline characteristics for the treatment 
and control groups, sample weights, the analytic approach, supplemental data, 
and estimates of ever attending a private school. (Contains 22 tables and 52 
endnotes . ) (SLD) 
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EXECUTIVE SUMMARY 



In 1997, the School Choice Scholarships Foundation (SCSF) annoiuiced that it would 
provide 1,300 scholarships so children of low-income families in grades K through four in New 
York City public schools could transfer to private schools. The scholarships were worth up to 
$1,400 annually and could be used for up to four years at both religious and secular schools. The 
SCSF received applications from more than 20,000 students between February and April 1997. 
From the pool of applicants, scholarship recipients were selected in a lottery held in May 1 997 . 

In this report, we describe the second-year results for an evaluation of the SCSF program. 
The evaluation is based on a rigorous research design that takes advantage of the fact that the 
SCSF held a lottery. Accordingly, the evaluation allows for the conduct of a randomized 
experiment in which students were randomly selected for a treatment group (scholarship group) 
and a control group. The results are particularly relevant to the debate about the impacts of 
education vouchers on students and parents. Among the current voucher experiments, the SCSF 
program is the largest in terms of enrollment and shows results for the most diverse population of 
low-income students. Similar randomized field trials of school voucher interventions have been 
conducted in Dayton, Ohio, and Washington, D.C. 

Key findings from the evaluation are as follows: 



OUTCOMES 

Impacts on School Facilities, School Climate, Parents’ Satisfaction with Schools, and 
Parental Communication and Involvement 



• As reported by parents, the schools attended by the scholarship students were smaller 
than the schools attended by the public school students (385 students versus 525 
students) and class sizes were smaller (two fewer students in the private school 
classrooms). Private schools were less likely than public schools to have a library, 
nurse’s office, child counselors, and special programs for non-English speakers and 
students with learning problems. In a few instances, private school parents reported 
more extensive facilities and programs, such as computer laboratories, music 
programs, and individual tutors. No differences were foimd concerning programs in 
art, programs for advanced learners, a gymnasium, and after-school programs. 

• Compared with public school parents, private school parents were less likely to report 
the following serious problems at their school: students destroying property, 
tardiness, missing classes, fighting, cheating, and racial conflict. For example, 70 
percent of the parents with a child in public school reported that fighting was a 
serious problem at their child’s school, as compared to 33 percent of the parents with 
a child in private school: 
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• Students in private schools were asked to complete more homework than students in 
public schools. 64 percent of the parents of children in private school said their child 
had more than an hour of homework per day, as compared to 41 percent of the 
parents whose child attended a public school. 

• Parents of students in private schools said that they received more communication 
from their school about their children than did parents in public schools. 

• The level of parent involvement for parents with students in private schools was 
about the same as for parents with public school children. 

• Compared with public school students, private school parents said their children were 
more likely to have received religious instruction outside of school, participated in 
church youth groups, and attended religious services. 

• Parents who switched from public to private schools were much more satisfied with 
their schools than parents who remained in the public schools — for example, when 
asked to grade their schools, nearly 40 percent of the parents in the scholarship takers 
group gave their school an A and less than 1 0 percent of similar parents in the public 
schools gave their schools an A. 

Impacts on Test Scores 

• On standardized tests, students offered scholarships (24 percent never used a 
scholarship when offered) generally performed at about the same level as students in 
the control group. That is, using the Iowa Test of Basic Skills to assess students’ 
performance in reading and mathematics, we found that, overall, students in the 
scholarship group performed at about the same level as did students in the control 
group. The same results hold for students who shifted from a public school to a 
private school. 

• To see whether the voucher intervention affected the often reported test-score gap 
between whites and minority students, results were examined for Latinos and African 
Americans separately. African American students make up about 45 percent of the 
sample and Latino students about 46 percent. The pattern of impacts for Latino and 
African American students is inconsistent. We find no impact for Latinos. We find a 
significant average impact on the composite test scores for African American 
students. African American students offered scholarships scored about 3 points 
higher than similar students in the control group. The impact of going to a private 
school for two years for African American students was 4 percentile points. 

• Much of the overall impact of a voucher on African American students’ achievement 
is concentrated among those students who were in 6*'’ grade. The impact for sixth 
graders is statistically significant and large; the impacts of a voucher on African 
American students’ achievement in grades 3-5 are much smaller and not statistically 
significant. 
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• A comparison of the first- and second-year results for African Americans showed 
similarly sized impacts. In the first year, the overall impact of a voucher offer on 
reading for African American students was 3.5 percentile points and for the second 
year 3.4 percentile points; this difference is not statistically significant. For 
mathematics, there was a slight decline in impacts — from 5.4 to 3.1 points; that 
decline is also not statistically significant. Taken together, these findings suggest that 
the difference between the scholarship group and the control group remained about 
the same over the two years. Similar to the findings for African Americans, there 
were no significant changes for Latino students between year one and year two. 

PARTICIPATION IN THE SCSF PROGRAM 

• About 62 percent of the students offered scholarships used the scholarship for two 
full years, 12 percent used them in just the first year and not the second year, 2 
percent used them only in the second year, and 24 percent never used them. 

• Students who used the scholarships were generally similar to nonusers, but there 
were some differences. Baseline test scores were similar for scholarship takers and 
decliners; households of scholarship takers and decliners were equally likely to speak 
English as their main language; and mothers of takers and decliners were equally 
likely to have been bom in the Lfnited States. Scholarship decliners were somewhat 
less likely than scholarship takers to have received special education services before 
the baseline testing session; mothers of scholarship takers had somewhat higher 
educational attainment than the mothers of the decliners; and scholarship takers lived 
in families with higher incomes than scholarship decliners (about $2,700 higher; the 
average income for takers was about $10,400). 

• The most frequently cited obstacles that prevented parents from sending their children 
to the preferred school included cost (35 percent), transportation problems (14 
percent), and lack of space at the school (10 percent). 

• According to parental reports, the percentage of students attending school throughout 
the school year was similar for those attending public and private school. Similarly, 
the percentage of students who plan to attend the same school the next year was 
similar for the two groups, except that public school students were more likely to 
graduate from one school level to the next (private schools are more likely to have all 
grades K-8 within one school). Virtually no parents reported their child had been 
expelled from school. Suspension rates for students in public and private school were 
similar. 

THE EVALUATION 

The evaluation of the SCSF program in New York City presents a unique opportunity to 
examine the impact of educational vouchers on student and parent outcomes for students 
switching to private schools. New York City has a diverse population and is the nation’s largest 
school system. We computed the effects of vouchers on education outcomes by using a 



randomized experimental design that allows us to compare two statistically equivalent groups of 
students and, in turn, to isolate the unique effect of vouchers on the measured outcomes. 

Mathematica Policy Research has collected data three times on the same students and 
families since 1997 (1997, 1998, and 1999). We have just completed collecting a forth round of 
data and will report the results early in 2001. Each time we collected data, students were given 
the Iowa Test of Basic skills to measure their academic achievement in reading and mathematics. 
In addition, parents and students completed surveys in each of the three years so that we could 
learn more about their educational experiences and plans. The response rate for each test 
administration was moderately high — 100, 78, and 65 percent in 1997, 1998, and 1999, 
respectively. Somewhat higher response rates were achieved for the parent and student surveys 
than for the achievement tests. 



SCHOOL CHOICE IN NEW YORK CITY AFTER TWO YEARS: 
AN EVALUATION OF THE SCHOOL CHOICE 
SCHOLARSHIPS PROGRAM 



Over the past few years, Congress and many state legislatures have introduced school 
voucher proposals. At the same time, three publicly — funded programs are currently in operation 
in Milwaukee, Cleveland, and Florida.' A larger number of privately funded voucher programs 
have also been funded — in New York City; Washington, D.C.; Dayton, Ohio; San Antonio, 
Texas; Indianapolis, Indiana; and San Francisco, California as well as in a nationwide program.^ 

In this report, we seek to add to the growing body of knowledge about the workings of 
school voucher programs by reporting the results from the second year of the operation of the 
voucher program serving students who had previously been attending New York City public 
schools. Begun in 1996 by the School Choice Scholarships Foundation (SCSF), this privately 
funded voucher program was designed to allow for the collection of high-quality information 
about student test-score outcomes and parental assessments of public and private schools. 
Taking advantage of the fact that scholarships were awarded by lottery, the evaluation was 
designed as a randomized field trial. Before conduct of the lottery, the evaluation team collected 
baseline data on test scores and family background characteristics.^ The evaluation is continuing, 
with results from the second year of the pilot program contained in this report. 

Many interest groups, political leaders, and policy analysts have debated the desirability of 
continuing and expanding school choice programs." Unfortimately, high-quality information that 
might inform this debate is limited. Although many studies comparing public and private 
schools have been published, they have come under criticism for comparing dissimilar 
populations. Even when statistical adjustments are made, it remains unclear whether findings 
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actually describe differences between public and private schools or simply differences in the 
types of students and families attending them.^ 

The best way to make sure that two populations are similar is to assign individuals randomly 
to treatment and control groups. Random assignment has been used recently in a number of 
educational studies, such as the Tennessee Star study that found that smaller classes had positive 
effects among students in kindergarten and first grade.^ Another example of random selection of 
students to treatment and control groups is the national evaluation of Upward Bound, which is 
funded by the U.S. Department of Education.^ Until recently, random assignment had not been 
used carefully to study the question of school choice. 

The SCSF program provides an opportunity to estimate the impacts of a school choice pilot 
program with the following characteristics: 

• a lottery that allocated scholarships randomly to applicants; 

• a lottery that was administered by an independent evaluation team; 

• collection of baseline data on student test performance and family background 
characteristics from a high percent of the students and their families before the 
lottery; and 

• data on a broad range of characteristics of parents and students collected from a 
reasonably high percent of the treatment and control groups two years after the 
beginning of the intervention. 

In this report, we provide information about the SCSF program two years after students 
started using their school-choice scholarships. Similar reports are planned for at least one more 
year of the program. In the remainder of this report, we describe the program sponsored by the 
SCSF; the data collection, analysis, and reporting procedures followed by the evaluation team; 
and detailed findings from the evaluation for the first year of the program. 
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DESCRIPTION OF THE SCHOOL CHOICE SCHOLARSHIPS FOUNDATION 
PROGRAM 

In February 1997, SCSF announced that it would provide 1,300 scholarships worth up to 
$1,400 annually for at least three years to children from low-income families currently attending 
public schools. The scholarship could be applied toward the cost of attending a private school, 
either religious or secular. After announcement of the program, SCSF received initial application 
forms from over 20,000 students between February and late April 1997. 

To be eligible for a scholarship, children had to be entering grades one through five, live in 
New York City, attend a public school at the time of application, and come from families with 
incomes that qualified them for the U.S. government’s free school lunch program. To ascertain 
eligibility, students and an adult member of their family were asked to attend verification 
sessions during which family income and the child's public-school attendance were documented. 

Because many more families applied for scholarships than originally projected, we 
randomly selected families for scholarships through a two-stage procedure. As families applied 
for scholarships, they were organized into groups on the basis of their application date. During 
the early stages, all families were invited to eligibility assessment and data-collection sessions. 
However, after it became clear that more families would be attending the sessions than could be 
accommodated, we began randomly selecting applicants, inviting only those selected to attend 
the sessions. After the first stage was completed, families who attended the sessions and met the 
eligibility requirements were then randomly selected for the scholarship or control group. To 
ensure that all families from the different sessions had the same chance of selection for the 
scholarship group, we adjusted the second-stage selection probabilities to reflect the differential 
chances of being invited to the verification sessions. 
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The final lottery was held in mid-May 1997. Mathematica Policy Research (MPR) 
administered the lottery; SCSF announced the winners. Within the parameters established by 
SCSF, all applicants had an equal chance of winning the lottery. SCSF decided in advance to 
allocate 85 percent of the scholarships to applicants from public schools whose average test 
scores were less than the citywide median. Consequently, applicants from these schools, who 
represented about 70 percent of all applicants, were assigned a higher probability of winning a 
scholarship. In the information reported in the tables, results have been adjusted by weighting 
cases differentially so that they can be generalized to all eligible applicants who would have 
attended the verification sessions had they been invited regardless of whether they attended a 
low-performing school. Subsequent to the lottery, SCSF assisted families in finding private 
school placements. 



EVALUATION PROCEDURES 

The evaluation procedures used for the second follow-up data collection were similar to 
those used during the baseline and first follow-up. Below, we describe these procedures in 
greater detail and present response rates for each of the data-collection components. 



Collection of Baseline Data 

During the eligibility verification sessions, students were asked to take the Iowa Test of 
Basic Skills (ITBS) in reading and mathematics. Students in kindergarten applying for a 
scholarship for first grade were exempted from this requirement. Parents were asked to fill out 
questionnaires reporting information on their satisfaction with the school their child was 
currently attending, their involvement in their child's education, and their demographic 
characteristics. These sessions took place during March, April, and early May 1997 on Saturday 
mornings and on vacation days. The sessions were held at private schools, where students could 
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take tests in a classroom setting. In most cases, private school teachers served as proctors under 
the overall supervision of the staff of MPR. 

While the child was taking a test that took more than an hour, the adult accompanying the 
child to the testing session responded to the questionnaire in a separate room. This procedure had 
the advantage of giving administrators the opportunity to stress that responses to the 
questionnaire would be held in strict confidence and used for statistical purposes only. It also 
provided respondents with the time to complete the questionnaire at a leisurely pace and the 
opportunity to ask any questions concerning the meaning of particular questions. Questionnaires 
were available in both English and Spanish. 

Given the likelihood that a variety of caretakers might be accompanying children, questions 

were designed to allow any caretaker familiar with the child’s school experiences to respond to 

the questions. Although grandmothers and other relatives and guardians also filled out the 

questionnaire, in over 90 percent of the cases one of the parents answered the questions. The 

remainder of the report, for ease of presentation, refers to opinions expressed as those of parents. 

Because scholarships were allocated by a lottery, there were few differences between those 

students offered scholarships and those who were not (see Appendix A). Those scholarship 

recipients who made use of the scholarship were more educated and somewhat less 

disadvantaged in other respects than those who did not make use of the scholarship; however, 

those making use of the scholarship were more likely to be Afiican American. Baseline test 

8 

scores did not differ significantly between those using the scholarship and those who did not. 

Collection of Second-Year Follow-Up Information 

To evaluate the effects of the scholarship on students and their families, MPR constructed 

two statistically equivalent groups of families: (1) a scholarship group with 1,000 families and 
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(2) a control group with 960 families. Procedures used to construct the two groups and to collect 
first-year follow-up information and the results from the evaluation of the first year of the 
program are described elsewhere.^ For the second-year follow-up, families were invited in April, 
May, and June 1998 to attend sessions during which students again took the ITBS in 
mathematics and reading. Adult members of the students’ families completed surveys that asked 
a wide range of questions about the educational experiences of their oldest child within the age 
range eligible for a scholarship. Students in grades three through six were also asked to complete 
short questionnaires. 

Testing and questionnaire administration procedures were similar to those followed during 
the baseline and first-year follow-up sessions. Both the scholarship students and students in the 
control group were tested in locations other than the school they were currently attending. 

Response rates for the second follow-up survey and test, along with response rates for the 
baseline and first follow-up surveys and test administrations are shown in Table 1. We present 
results separately for the treatment and control groups and distinguish between students who did 
not attend testing sessions and those who completed too few items on the reading and 
mathematics tests to be scored by the test publisher. Riverside Publishing. Seventy-four percent 
of the families selected for inclusion in the evaluation participated in the sessions held in spring 
1998. This fairly high response rate was achieved in part because SCSF conditioned the renewal 
of scholarships on participation in the evaluation; nonscholarship winners selected to become 
members of the control group were compensated for their expenses and told that they could 
automatically reapply for a new lottery if they participated in the follow-up sessions. 

As seen in Table 1, the participation rate was fairly similar for treatment and control groups; 
75 percent of the families offered scholarships participated in the evaluation compared with 72 
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percent of the families in the control group. Sixty-five percent of the students participating in the 
evaluation attended the testing session, including 69 percent of those offered scholarships 
compared with 60 percent of the students in the control group. Seventy-nine percent of the 
students were scored on the baseline reading test and 73 percent on the mathematics test. The 
percent of completed student tests is less than the family survey response rates because some 
students did not complete a sufficient number of items to have their tests scored. Incomplete tests 
could have resulted for a variety of reasons. For example, some students had never taken a 
standardized test before we administered one to them and found the experience overwhelming; 
these students were dismissed from the testing session. A few other students decided for one 
reason or another that they did not want to complete the test items. In a few cases, the test 
proctors took too long to administer the tests and that the session concluded before the children 
had time to complete a section of the test. 

Although the background characteristics of participants and nonparticipants in the second- 
year follow-up, as observed in the baseline survey conducted in 1997, resembled one another in 
most respects, they differed significantly in some (see Appendix A). Among the treatment group, 
participants were less likely to be white, less likely to be black, and more likely to be Hispanic 
other than Puerto Rican. Mothers were more likely to be bom outside the United States, more 
likely to have lived in the same residence, less likely to be working, more likely to state their 
religious affiliation as Catholic, and less likely to use food stamps or welfare. They originally 
reported an average income of around $9,900 compared with $8,500 for nonparticipants. They 
were less likely to speak English at home. The student was less likely to have been in third grade 
at the time of the initial application for a scholarship. 
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Members of the control group who participated in the second-year follow-up were less likely 
than nonparticipants to be black and more likely to be Hispanic other than Puerto Rican. They 
were more likely to report that their child had received help for a disability. They were more 
likely to note a Catholic religious affiliation. They were more likely to be receiving supplemental 
security income and less likely to speak English at home. Students were more likely to have been 
first graders at the time of the initial application for a scholarship. 

To adjust for survey nonresponse in our statistical analyses, we use an analytic model to 
predict nonresponse based on a variety of background characteristics. The predicted probability 
of not responding is then used to adjust the sample weights. A more detailed discussion of this 
procedure is described in Appendix B.'° 

Data Analysis and Reporting Procedures 

The analysis of the data from the second-year evaluation of the SCSF program takes 
advantage of the fact that a lottery was used to award scholarships. As a result, it is possible to 
compare two groups of students that were similar, on average, in almost all respects except that 
the members of the control group were not offered a scholarship. 

This report provides data that help answer the following two questions, both of which have 
clear policy implications; 

1. What was the impact of the ojfer of a SCSF scholarship to a group of low-income 
scholarship applicants, as measured by test scores and as perceived by the applicants 
themselves? 

2. What was the impact of attending a private school? 

The analytical techniques needed to answer each question differ in important ways. The first 
question can be answered straightforwardly by comparing the responses of those who were 



offered a scholarship with the responses of the control group. To compute the impact of a 
scholarship offer on children’s test scores, we estimated statistical models that take into account 
whether a student was offered a scholarship as well as baseline reading and mathematics test 
scores and variables that define the randomization process. Baseline test scores were included to 
adjust for chance differences between the scores of treatment- and control-group members on the 
achievement tests and to increase the precision of the estimated impacts. We used a similar 
approach to compute the impacts of the program on the parent and student survey responses. In 
equations predicting parent and student responses, we did not, however, include baseline data, 
other than a treatment indicator and variables used to define the randomization process. 

To compute the effects of going to a private school on students’ test scores, a statistical 
model was estimated that took into account test scores and variables used to define the 
randomization process as well as whether students attended a private or a public school. To 
compute program impacts on parent and student survey items, we used a similar approach; 
however, we did not include the baseline test scores to predict parent and student responses. 

The Appendix presents a detailed description of the model estimation procedures and 
procedures used to compare results from year 1 with year 2 are presented in the Appendix C. 

The answer to the first question — the impact of an offer of a scholarship— is provided in 
column three of many of the tables in the main body of the report. It is the difference between 
column one, the response of those offered a scholarship, and column two, the response of those 
not offered a scholarship. The answer to the second question, the impact of going private, is 
provided in column six. It is the difference between the estimated response of those attending a 
private school in columns four and five, which provide an estimate of the response of the 
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appropriate control group. 



For some policy analysts, the first question is the crucial policy question: What happens 
when a school choice program is put into effect? What are the impacts on the population of low- 
income families interested in a school-choice scholarship? This is similar to a question often 
asked in medical research: What will happen if a particular pill is marketed? How will the health 
of potential users be altered whether or not all patients use the pill as prescribed? 

For other analysts, the second question is critical because it tells us about the potential 
impacts of attending private instead of public schools. They want to know what are the 
consequences of actually attending a private school and not just whether an offer was made. 
More exactly, analysts want to know what difference it makes whether low-income, inner-city 
families attend a public or a private school. In medical research, the parallel question is: What 
are the consequences of actually taking a pill as prescribed? 

In short, when we provide information that answers the two questions enumerated above, we 
present the information in six columns as follows: 

1 . Response of all those offered a scholarship; 

2. Response of those not offered a scholarship; 

3. Estimated impact of being offered a scholarship; 

4. Responses of all those who attended private school; 

5. Estimated response of those in appropriate control group; and 

6. Estimated impact of attending a private school. 

A CONTEXT FOR INTERPRETING PROGRAM IMPACTS 
Response Bias 

The interpretation of data from the parental and student surveys in New York City needs to 
take into account parents and students exaggerating their responses to some items, such as 
satisfaction with their schools, time spent on homework, or educational expectations. No special 
weight should be placed on the actual frequency with which any particular type of event is said 
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to take place. For example, one should not take too seriously the claim by children in third 
through fifth grades that they spend, on average, approximately one hour and 20 minutes a day 
on their homework. 

But if absolute levels may not be estimated accurately, there is no reason to believe that the 
two groups of parents — scholarship recipients and members of the control group — differ in the 
accuracy of their reports. After all, individuals were assigned randomly to the two groups, and 
any reporting bias should be similar for the two groups. 

Thus, this report will, for the most part, interpret differences between groups rather than the 
absolute value obtained by any one group. 

Generalization of Findings 

It is essential to qualify any generalizations from the results of this pilot program to a large- 
scale voucher program that would involve all children in New York City or other central cities. 
Only a small fraction of low-income students in New York City public schools were offered 
scholarships, and they constituted only a small proportion of the students attending New York 
private schools. The impact of a much larger program could conceivably have different program 
outcomes. 

Although one cannot generalize the findings to programs that involve many or all students 
within a school district, the results from an evaluation of the SCSF program may permit 
estimation of the likely impact of a small-scale publicly funded voucher program serving low- 
income families. Earlier research indicates only modest differences in the family background 
characteristics of those eligible to receive an SCSF scholarship and actual scholarship 
applicants.'^ No significant difference was observed in the income of applicants compared with 
the income of eligible population. Fathers’ employment rates were similar. In addition, the 
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residential mobility of the applicant population was about the same as among the eligible 
population. And applicant mothers were only slightly more likely to be foreign bom than 
mothers in the eligible population. 

In some ways, applicants seem to have been more disadvantaged than eligibles. Applicants 
were more likely to be dependent on government assistance for income. In addition, the applicant 
population was less likely to be non-Hispanic white and more likely to be African American. At 
the same time, however, other findings point in the opposite direction. Mothers and fathers are 
considerably more likely to have some college education; English was more likely to be the 
language spoken in the household; and mothers were more likely to be employed either fiill or 
part-time. 

Furthermore, any voucher program directed at low-income families is likely to attract 
initially those families with the greatest interest in exploring an educational alternative, precisely 
the group that applied for an SCSF scholarship. Because such a group may be more select, it is 
not known whether the results from the evaluation would generalize to those families who might 
later apply for scholarships. 



SCSF Program Participation and Members of the Control Group Attending Private 
Schools 

The number of children who took advantage of a scholarship offer is an important factor in 
interpreting the impacts of a scholarship offer because it provides some indication of the strength 
of treatment. Before the 1997-1998 school year, SCSF offered scholarships to 1,374 children. 
By the end of the second year, about 64 percent of these children were using a scholarship: 62 
percent of the children had used a scholarship for two full years, 1 2 percent used one just in the 
first year, and 2 percent used a scholarship only in the second year. Most families who decided 
not to use a scholarship based their decision on financial reasons, recognizing that the $1,400 
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scholarship does not cover the full cost of tuition. (Later in this report, we provide a more 
detailed analysis of the reasons given by parents for both selecting specific schools and leaving 
the program.) 

Besides taking into account the number of students in the treatment group who used their 
scholarships, one also must consider the behaviors of the control group; that is, to what extent 
did families in the control group send their children to private schools. Starting with the first 
follow-up survey, we asked parents in the control group whether their children were attending a 
private school. We discovered that about 8 percent of all children in the control group were 
reported as attending a private school for at least one year: 4 percent attended both years and 4 
percent attended for one year. 

If all children randomly assigned to the scholarship group had attended a private school and 
all children in the control group had attended a public school, then the treatment differential 
would have been 100 percentage points. The private school attendance patterns of the treatment 
and control groups suggest that the treatment differential is about 58 to 64 percentage points.'^ 
The estimated impact of a scholarship offer takes into account both the impact on those who use 
the scholarship, and the proportion of those offered the scholarship who do not make use of it, 
and the proportion of the control group that attended a private school. In other words, observed 
effects of being offered a scholarship are approximately 60 percent of the actual effects of 
attending a private school because only 60 percent of those offered the scholarship complied 
with the protocols. For this reason, in the pages and tables that follow, the estimated impacts of 
actual attendance at a private school are always substantially larger than the estimated impacts of 
a scholarship offer. 
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Participation in Scholarship Program 

An important issue in the school-choice debate concerns the composition of those who 
would leave public schools if school vouchers were made generally available. Critics of school 
choice have argued that choice programs do not give low-income families a viable choice of 
schools. In the words of educational sociologist Amy Wells, “White and higher-SES 
[socioeconomic status] families will no doubt be in a position to take greater advantage of the 
educational market.”''’ The president of the American Federation of Teachers (AFT), Sandra 
Feldman, has claimed that vouchers for private schools take "money away from inner city 
schools so a few selected children can get vouchers to attend private schools, while the majority 
of equally deserving kids, who remain in the public schools, are ignored."'^ But the first-year 
evaluation of the SCSF program, as well as the evaluation of other voucher programs in 
Cleveland and San Antonio, indicated that those who made use of a voucher did not differ 
sharply from those who did not.' ^ 

Still, these were initial results after one year. Even if voucher programs are only modestly 
selective in the initial year, does a significant degree of selectivity become apparent after two 
years? 

Given that 62 percent of those offered a scholarship used the scholarship for two full years, 
that 12 percent used it in just the first year, and that 2 percent used the scholarship only in the 
second year, it is now possible to ascertain the degree of selection after two years by comparing 
the initial characteristics— as observed at baseline— of those using the scholarship two years later 
with the characteristics of those offered scholarships but not using them two years later. As seen 
in Table 2, little selection on education criteria had taken place by the end of the second year. 
Those offered a scholarship showed no significant initial test-score differences compared with 
those who made use of the scholarship. For those still making use of the scholarships two years 
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later, initial reading scores collected during the 1997 baseline testing session averaged only a 23 
national percentile point ranking (NPR), the same as those not making use of the scholarship two 
years later. For mathematics, the difference was only one NPR, not a statistically sigmficant 
difference. Those who did not make use of the scholarship were slightly more likely to have 
received special education services before the baseline testing session— the difference was 4 
percentage points — 14 percent of the decliners compared with 10 percent of those still making 
use of the scholarship two years later. In addition, households of scholarship takers and decliners 
were equally likely to use English as their main language. Mothers of decliners were also no less 
likely to have been bom in the United States. 

Demographically, other differences were apparent. For example, mothers of students 
remaining in the scholarship program two years later were more likely to have more than a high 
school education. In particular, 57 percent of mothers of children using the scholarship had more 
than a high school education compared with 48 percent of mothers whose children did not use 
the scholarship. Those who took advantage of the scholarship were also more likely to have lived 
at their current residence for two years or more. 

Economically, scholarship users were in better circumstances than nonusers. The reported 
family income of scholarship takers was about $2,700 higher. Mothers were more likely to be 
working full time, less likely to be on welfare. Takers were also somewhat more likely to be 
Catholic — 54 percent of scholarship users compared with 46 percent of the nonusers. 

While some who declined the scholarship may have done so because they decided that the 
public schools better suited their needs, most parents felt otherwise. During the second year of 
the study, only 48 percent of those who declined the scholarship, compared with 88 percent of 
the scholarship participants, reported that their children attended a school they preferred. The 
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most frequently cited obstacles to sending children to a preferred school included cost (35 
percent), transportation problems (14 percent), and lack of space at the school (10 percent). That 
cost was cited most frequently is not surprising given that the $1,400 voucher does not cover the 
full tuition and expenses at private or parochial schools. The median tuition, according to those 
attending private schools, was $2,000, and the median additional expenses for imiforms, school 
activities, books, supplies, and related items was $500. Therefore, most families who accepted an 
SCSF scholarship needed to find approximately $1,100 per child in supplemental fiinds.'^ 

Still, we are left with the question of where scholarship families obtained additional 
resources. After the first year, we asked scholarship parents how they paid the tuition and 
additional expenses, inviting them to list more than one source of revenue, if appropriate. The 
most frequently mentioned source of funds was family income (34 percent of the scholarship 
users). Twenty-two percent of the families that used a scholarship in the first year said their child 
had received a school scholarship, and 5 percent said the school paid for some or, in a few cases, 
all of the tuition. Sixteen percent of the respondents said that relatives and fiiends helped out. 
Only 4 percent said they paid for tuition by donating time and fund-raising support to the school. 

Selecting a School 

Voucher critics often disagree with proponents of school choice about the importance of 
educational considerations in the selection of the school. Critics argue that low-income families 
are more concerned about location, sports programs, or religious instruction than about academic 
quality per se. For example, the Carnegie Foundation for the Advancement of Teaching has 
claimed that "when parents do select another school, academic concerns often are not central to 
the decision."'^ Similarly, an American Federation of Teachers’s report on the Cleveland 
voucher program suggests that parents sought scholarships not because of "'failing' public 
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schools" but "for religious reasons or because they already had a child attending the same 
school."'^’ Disputing these contentions, supporters of school choice claim that low-income 
parents, like other parents, place the highest priority on the educational quality of the school. 

Parents were asked to select the three most important among a list of considerations they 
may have had in mind when selecting a school. Of those parents who took advantage of the 
scholarship, they most frequently mentioned academic quality (listed by nearly 60 percent of the 
parents; Table 3) while other considerations included school discipline (42 percent of parents), 
safety (36 percent), and teacher quality (35 percent). A little over a quarter listed religious 
instruction and what is taught in class. Twenty percent mentioned class size and a convenient 
location. Considerations mentioned by only a small fraction of the parents included the school 
facilities, the sports program, and the school attended by the child's friends. 

In sum, educational considerations seemed predominant, questions of social order 
(discipline and safety) secondary, and religious instruction of tertiary importance. 

Obtaining the School of Choice 

If offered a scholarship, parents were more likely to send their children to a school they 

preferred. Nearly 80 percent of those offered a scholarship reported success in finding a school 

2 1 

they wanted compared with 56 percent of the control group (Table 4). 

Parents who did not obtain the school of their choice were asked to identify why they 
suspected that their choice went unfulfilled. Parents could list more than one reason if they so 
wished. By reducing the cost of attending a private school, the scholarship reduced from 31 to 12 
percent the proportion of parents who said they could not afford their preferred school (Table 4). 
The cost issue notwithstanding, parents most frequently mentioned the following impediments 
(in order of frequency): no space available (5 percent of the parents); applied too late; 
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transportation problems; child not given space; school location; child failed the admission test; 
family did not share the school's religious affiliation; and the family moved away from the 
school. 



Experiences in School 

The type of school experience a child will enjoy as the result of a voucher intervention has 
been a matter of considerable debate. Choice critics say that public schools have better facilities 
and more elaborate programs capable of serving a diverse population^^ and that choice will lead 
to ethnic and racial segregation.^^ Choice supporters claim that private schools have the 
necessary facilities and do a better job of incorporating all children into a common framework 
and that the private sector is more integrated than the public sector.^^ Critics say that many 
private schools do not give students the necessary freedom to develop broadly while supporters 
say that privately run schools are more orderly, making it easier for children to learn. 

To address these issues, we provide information on school facilities, programs, ethnic 
composition, and the disciplinary climate in public and private schools. 

School Facilities. Most observers expect to find in central-city public schools larger, more 
expensive, more complex, and more sophisticated facilities than are available in central-city 
private schools. With a few exceptions, reports from applicant parents in New York City are 
consistent with the conventional wisdom. 

First of all, public schools are larger. As estimated by parents, the effect of choosing the 
private sector was to reduce the average size of the school by 140 students or over 25 percent — 
from an average of 525 students to 385 students (Table 5).^'* This estimate of the impact of the 
program is almost exactly the same as parents provided one year earlier as part of the first-year 
evaluation.^^ 
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Private schools were also less likely to have a library, a nurse’s office, child counselors, and 
special programs for non-English speakers and students with learning problems. The greatest 
difference was for programs for non— English speaking students. Forty- four percent of the 
private-school parents reported such a program in their school compared with 80 percent of the 
control-group parents. Most other differences were not as large; for example, 58 percent of the 
private-school families reported that their school had a program for the learning disabled 
compared with 74 percent of the parents in the control group (Table 5). 

In a few instances, private-school parents reported more extensive facilities and programs. 
For example, they were somewhat more likely to say their school had a computer laboratory, a 
music program, and individual tutors. In other cases such as programs in arts, programs for 
advanced learners, a gymnasium, and after-school programs, no differences between the two 

groups were evident. The reports from parents concerning school facilities were similar to those 
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reported at the end of the first year. 

As at the end of the first year in the program, private- school parents reported at the end of 
the second year that their children had been in smaller classes.^’ The effect of using a scholarship 
was to reduce the size of the child’s class by two students (Table 5). 

In sum, classes and schools are smaller in the private sector, but public schools offer a wider 
range of facilities and programs. The larger, more complex facilities do not, however, seem to 
satisfy the parents who applied for scholarships. On the contrary, approximately 33 percent of 
those with students in private schools were very satisfied with school facilities compared with 5 
percent of the parents in the control group (Table 5). 



Ethnic Composition of School. Shifting to a private school in New York City somewhat 

reduced the racial isolation of minority students. Parents were asked, “What percentage of the 

students in this child’s classroom are minority?” To this question, 38 percent of control- group 

members replied that everyone in the classroom was of minority background (Table 6). Only 30 

percent of the private-school parents gave the same response. The results are similar to those 

28 

reported in the first-year evaluation. 



Special Education. In the debate over school choice, special education has received a good 
deal of attention. Critics of school choice say that private schools ignore the needs of those with 
physical and mental disabilities.^^ Defenders of school choice often claim that many of those 
diagnosed as disabled can learn in regular classrooms and that special arrangements can be made 
for others. 

To illuminate this question, parents were asked about their child’s special education needs 

( 

and the availability of school programs to meet those needs. The number of learning disabled 
and physically disabled students in this evaluation was small, however; as a result, the 
differences in parents’ assessments of school performance, though fairly large, are not 
statistically significant. Nonetheless, given that the differences are fairly large, they are worth 
reporting but should be interpreted with caution. 

Only 4 percent of those offered a scholarship said that their child had a physical disability, 
and just 10 percent said their child had learning difficulties (Table 7). Of those with learning 
disabilities, scholarship users were more likely to say that the facility met the child’s needs very 
well. Specifically, nearly 26 percent of the private-school parents said that the schools were 
meeting the learning needs very well compared with 16 percent of the control-group parents. 
However, parents of public-school students with physical disabilities were more likely to say that 
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their school met students' needs. Sixty-two percent of the public-school parents, but only 33 
percent of those with a child in private school, reported that the school did "very well" at meeting 
their child's needs. 

In sum, it seems that private schools may be better able to meet the needs of the learning 
disabled while public schools command the facilities and resources for better meeting the needs 
of the physically disabled. Given the small numbers involved, the findings are tentative. 

School Climate. If parent reports are accurate, the scholarship program had a major impact 
on the daily life of students at school. Private-school parents were more likely to report that the 
following were not a serious problem at their school: students destroying property, tardiness, 
missing classes, fighting, cheating, and racial conflict. For example, 33 percent of the private- 
school parents thought that fighting was a serious problem at their school versus 70 percent of 
the control group (Table 8). The percents perceiving tardiness as a problem were 33 for the 
scholarship users and 62 for the control group. Less than 30 percent of private-school parents but 
45 percent of the control group said that destruction of property was a serious problem at their 
school. The results are similar to those reported in the first-year evaluation.^*^ 

Although student reports of the climate in their school and classroom are not as sharply 
differentiated as those of parents, they are consistent with parental assessments. As seen in 
Table 9, students in private school were more likely than the control- group students to report 
that students "get along with teachers" and are less likely to say that "teachers ignore cheating" 
or that "there is a lot of cheating in this school." They were also more likely to report that they 
had close friends who "got good grades" and less likely to report friends who "use bad 
language." 
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As reported in the first-year evaluation report, public and private schools seem to use 
different control mechanisms for maintaining discipline.^' Private schools seem to emphasize 
dress and orderliness; public schools rely on rules and regulations. Almost all private schools 
seem to require students to wear a school uniform. No less than 96 percent of the parents 
reported that their private school required uniforms compared with 43 percent of the parents in 
the control group (Table 8). Similarly, 95 percent of the private- school parents reported that 
certain types of clothing are forbidden, but less than two-thirds of the control group do. On the 
other hand, parents report that public schools more frequently employ sign-in sheets and hall 
passes. Ninety-five percent of the control group reported that parents must sign-in when they 
come to school, but just 86 percent of the private-school parents reported such a regulation. To 
leave their class, control-group students must obtain a hall pass, according to about 86 percent of 
the control-group parents, but only about 74 percent of the private-school parents mentioned a 
similar requirement. 

Homework 

After two years, parents continue to say that students in private schools are asked to do more 
homework.^^ Sixty-four percent of private-school parents reported that their child had at least an 
hour of homework a day, whereas only 41 percent of the control-group parents reported a similar 
volume of homework (Table 10). Private-school parents were also less likely to say the 
homework was too easy. Twenty percent of the control-group parents gave the same response 
compared with 4 percent of private-school parents. 

Student assessments of their homework were not as sharply differentiated as those of 
parents, but the differences were in the same direction. Students attending private school 
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estimated that they spent, on average, 51 minutes per typical night on homework compared with 
a 44-minute estimate reported by control-group members (Table 10). 

In one respect, student reports concerning homework differ significantly between the first 
and second years. After one year, students new to private schools were more likely than control- 
group students to report difficulty in keeping up with their homework.^^ After two years, the 
difference was no longer apparent (Table 10). Students were adjusting to the homework 
expectations of their new school, or the school was adjusting to the new students. 

School Communications with Parents 

Compared with control-group parents, parents of students in private schools said that they 
received more communication from their school about their child. Responses to questions about 
parent-school communications were very similar to those reported in the first-year evaluation. 
Although no significant differences in the frequency of parents’ nights were reported, the data 
presented in Table 1 1 indicate that a higher percent of private-school parents versus control- 
group parents reported 

• being more informed about student grades halfway through the grading period; 

• being notified when their child is sent to the office the first time for disruptive 
behavior; 

• parents speaking to classes about their jobs; 

• regular parent-teacher conferences; 

• parents participating in instruction; 

• parents receiving notes about their child from the teacher; 
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• parents receiving a newsletter about what is going on in school; and 

• regular parent-teacher conferences. 

The largest differences in school communication practices involved parents receiving 
newsletters, parents receiving notes about disruptive behavior, parents participating in 
instruction, parents receiving notes from teachers, and parents speaking about their jobs. For 
example, nearly 90 percent of the scholarship users compared with just over three-fourths of the 
control-group parents reported receiving notes from teachers. 

Religious Practices 

The SCSF program had an impact on students’ religious practices. Compared with the 
students in the control group, private-school parents more often said that their children received 
religious instruction outside of school, participated in church youth groups, and attended 
religious services (Table 12). Twenty-seven percent of the private-school students, but only 14 
percent of the control group, said that they have been receiving religious instruction outside of 
school. Half the students in private schools said they participated in church youth groups 
compared with 40 percent of students in the control group. Finally, students in private schools 
had attended church services more frequently than members of the control group. Nearly 60 
percent of the scholarship students reported attending religious services, as compared to less than 
a third of those in the control group. The effects of the scholarship program on student reports of 
their religious practice are as large after two years as they were after one year.^^ 

The higher level of religious activity among scholarship users was, in all likelihood, a 
genuine program impact, not a function of any selectivity in the population using the scholarship. 
The award of a scholarship was random, and two years earlier, when parents were asked in the 
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baseline survey about their religious affiliations, no significant differences in religious affiliation 
between the two groups could be detected. 

PARENTAL INVOLVEMENT IN CHILD'S EDUCATION 

Supporters of school choice claim that when parents choose a school, the family becomes 
more engaged in the child’s education. Working together, schools and parents create a more 
effective educational environment for their children.^^ But choice critics argue that any observed 
differences in parental engagement with private schools is due to the selected nature of the 
families that choose private schools in the first place. 

The evidence after two years provides little indication that school choice increases family 
engagement in children’s education. Nor has the program yet had any significant impact on 
parental involvement in children’s education (Table 13). Parents were asked how often they 
helped their child with homework, talked with their child about school, attended school 
activities, and worked on school projects. In every case, the answers given by the scholarship 
users and members of the control group were largely the same. These findings are similar to 
those reported in the first-year evaluation.^’ 

STUDENT ADJUSTMENT TO CHOICE SCHOOLS 

At least according to their survey responses, private-school students do not seem to have 

serious problems adjusting to their new classmates. As also observed at the end of the first 
year,^* private-school students reported the same average number of fnends in schools as did the 
control-group students. And students attending private school were no more likely to say that 
they often “feel made fun of’ by other students than were the control-group students, further 
evidence of adaptation to the new school. 
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Parental and Student Satisfaction 

Most studies of scholarship or voucher programs for low-income minority families have 
found that families receiving the scholarships are much more satisfied with their schooling than 
are families who remain in public schools.^^ The results from New York's second year confirm 
the earlier findings. When asked to assess their school overall, families give higher marks to the 
private schools. Nearly 40 percent of the scholarship users give their school an “A” compared 
with less than a tenth of the control group do (Table 15). 

We also examined parental satisfaction with specific dimensions of school life. On every 
aspect of a school about which parents were questioned, private-school parents were 
substantially more satisfied than control-group parents. The percent of parents “very satisfied” 
with a private school was significantly higher for all of the following: location of the school, 
school safety, teaching, parental involvement, class size, school facility, student respect for 
teachers, teacher communication with parents with respect to their child’s progress, extent to 
which child can observe religious traditions, parental support for the school, discipline, clarity of 
school goals, staff teamwork, teaching, academic quality, the sports program, and what is taught 
in school (Table 15). 

Forty-four percent of the private- school parents were very satisfied with the academic 
quality of the school as contrasted with just 5 percent of the control group. Similarly, 47 percent 
of the private-school parents expressed the highest satisfaction with “what’s taught in school” 
compared with 7 percent of the control group. 

The scholarship program had the smallest impact on parental satisfaction with schools’ 
sports programs. Less than a quarter of the scholarship parents were very satisfied with a 
school’s sports program compared with 6 percent of the control- group parents. 
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Differences in student reports of satisfaction were in the same direction but not as great as 
those reported by parents. In the short questionnaire administered to those in third through sixth 
grades, students were asked to give an overall grade for their school. The data in Table 15 
indicate that students in private school were more likely to give their school an "A" and less 
likely to give failing grades of “D” and “F.” Student reports after year two are very similar to 
those after year one. 

It may be hypothesized that the voucher program, like other innovations, has a Hawthorne 
effect, namely, the fact of innovation and change by itself enhances levels of parental 
satisfaction. If so, then the scholarship program might be expected to have a lesser impact on 
parental satisfaction after two years than after one. Parents may initially be impressed with the 
fact that they have a choice of school; with the passage of time, however, the initial impression 
may be moderated by the discovery that the school may not fully live up to its reputation. 

Differences in the level of satisfaction between public and private schools did not, for the 
most part, change significantly between the first and second years of the voucher program. On all 
the specific dimensions of school life about which parents were asked, differences in the percent 
of public- and private-school parents claiming satisfaction did not change significantly from one 
year to the next. For example, the difference in private- and public-school parent satisfaction 
with the academic quality of the child's school was 40 percentage points at the end of the first 
year and 39 points at the end of the second— a statistically insignificant change. Nor were there 
any statistically significant changes from the first to the second year in parental responses to the 
16 questions probing about satisfaction with other dimensions of school life, including class size, 
discipline, school safety, teaching, teacher-parent communication, and teaching values. 
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When, however, parents were asked the overall grade they would give the school attended 
by their child, we found some evidence that the program’s impact was declining. Whereas 49 
percent of private-school parents gave their school an "A" at the end of the first year, only 38 
percent did so at the end of the second year. Public-school parents who gave their school an "A" 
declined by only one percentage point, from 10 to 9 percent. In other words, the difference in 
private- and public-school parent willingness to give their school an "A" declined from 39 to 29 
percentage points, a significant change. Still, after two years, private-school parents were still 
much more likely to give their school an "A" than were control-group parents. 

CONTINUING IN THE PROGRAM 

It is generally thought that students perform better if they can remain in the same school 
throughout the school year and from one year to the next. Does school choice destabilize a 
child’s educational experience? In his evaluation of the Milwaukee school choice program, John 
Witte said that one of his concerns was the high rate of attrition from private schools.'*® And a 
number of choice critics have raised questions about the readiness of private schools to expel 
students who do not “fit in.”^' But other studies have found that students from low-income 
families are more likely to remain in the same school throughout the school year and from one 
year to the next.'*^ 

The SCSF pilot program provides an opportunity to examine this question after two years of 
student participation in a voucher program. In general, the findings confirm the conclusion that 
school choice does not disrupt the education of low-income students. 
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Suspension Rates and School Changes During the School Year 

As observed during the first year, suspension rates in the second year were much the same 
for both groups. Two percent of the parents in the control group and 3 percent in the private- 
school group reported that their child had been suspended (Table \A)P 

A very high percent of all students in the study remained in the same school for the entire 
year, much higher than is typical of inner-city minority children in general. The likely reason is 
that the families who applied for scholarships were strongly committed to their children’s 
education. No differences in school mobility rates are apparent between the two groups, a 
repetition of the finding observed in the first-year evaluation."*"* 

In short, school mobility was very low and virtually identical for both scholarship users and 
members of the control group. School expulsion or suspension was a trivial factor, affecting less 
than 1 percent of each group. These findings from the second-year evaluation resemble closely 
those observed after one year."*^ 



Plans for Next Year 

Scholarship recipients say they are more likely to attend the same school next year than are 
the members of the control group. More than 80 percent of the families of students attending 
private school said they expect their child to be back at the same school compared with about 60 
percent of the control group (Table 17). However, 17 percent of the control-group parents gave 
"graduating" as the reason for the change in schools, with only 2 percent of the scholarship 
students. Apparently, many of the students in public schools "graduate" from elementary to 
middle school, whereas students in private school do not. Once this difference in the organization 
of the school system is taken into account, there seems to be no significant difference in mobility 
rates from one year to the next for the two groups of families. 
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Approximately 4 percent of scholarship parents said they were changing schools because 
they did not find the quality of the school acceptable, and another 4 percent said they were 
pla nnin g to move away from the school. The next most frequently mentioned reasons for 
changing schools, given by no more than 3 percent of scholarship parents, were inconvenient 
location and school expense. None of the parents of students attending private schools said they 
had been asked by their school “not to return.” 

The reasons control-group parents gave for moving were fairly similar. Apart from 
graduation, the most frequently given reason was school quality. Seven percent of all control- 
group families said the quality of their school was not acceptable. Findings after two years do not 
differ in any important way from those observed after the first year.'*^ 

TEST PERFORMANCE 

This second-year evaluation of the SCSF program in New York City provides an 
opportunity to estimate the impacts on student test performance after a voucher program has 
been in place for two years. We report the impacts on test performances on the Iowa Test of 
Basic Skills in reading and mathematics in terms of (1) an offer of a school voucher and (2) 
going to a private school by the voucher recipients. Since mathematics and reading test scores 
were highly correlated, we report results for the combined performance of students on both tests 
along with the results for each test separately. The impact of a voucher offer is reported as the 
effect on student national percentile rankings (NPR), which may vary between one and 100. 
Nationally, the median NPR score on the Iowa Test of Basic Skills is 50. We also report effect 
sizes of selected impacts. 
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Results 



For all students in grades three through six, Tables 18 through 20 report the average impact 
of the offer of a voucher on the students’ combined test-score performance and separate 
estimates on their reading and mathematics scores. The tables also show average impacts for 
African American and Hispanic students.''^ The numbers of students from other ethnic groups 
were too few to permit separate analysis. To see whether the voucher intervention affected the 
often reported test score gap between minorities and whites, results are examined for Latinos and 
African Americans separately. 

Besides reporting results for all children or by race/ethnic groups, we report results 
separately for students in grades three, four, five, and six. Students are classified according to the 
grade they were expected to be in at the end of the second year of the voucher program. Most but 
not all students were in fact in the designated grade; some were held back a grade while others 
skipped a grade. To facilitate accurate comparison, all students were tested as if they were in the 
expected grade. Because baseline test scores were not collected from applicants in kindergarten 
at the time of application, no results are reported for these voucher students. 

The impact of a voucher offer combines results for both those who made use of the voucher, 
those who were offered a voucher but remained in public schools, and students in the control 
group who attended a private school, thus masking the potential impact of going to a private 
school. Using a statistical approach described in the appendix, we have estimated the impact of 
going to a private school and present the results alongside the impact of a voucher offer. Column 
seven reports the average programmatic impact of going to a private-school on student NPR 
scores. 
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Impact of a Voucher Offer 

Results from the SCSF evaluation show that after two years the offer of a voucher had no 
overall impact on student test performance (Tables 18 through 20); that is, students offered 
vouchers had about the same test scores as students in the control group. When we estimate 
impacts separately for Afncan American and Hispanic students, as for the year-one results for 
the evaluations in Dayton and Washington, a different picture emerges. That is, we find positive 
impacts for Afncan Americans but no impacts for Hispanic students. 

The combined test-score performance of Afncan American students who received a voucher 
offer was 3.3 NPR points higher than the combined test-score performance of those not offered a 
voucher (effects size = .16 of a standard deviation).^* The reading test-score performance of 
those offered a voucher was 3.4 percentile points higher and the mathematics 3.1 points higher 
(effect sizes are .16 and .13 of a standard deviation, respectively). The differences in combined 
test-score performance for the voucher group and the control group was statistically significant. 
The reading-score differences for the groups were also significant, but the mathematics-score 
differences, though nearly as large, were not statistically significant. 

When we estimate impacts separately by grade level for all students, we find no impacts on 
the composite test scores for students. A look at the two parts of the composite score reveals a 
significant impact on reading achievement in grade six (effect size = .21). We find no impact by 
grade level on mathematics achievement. Among African Americans, we observe a moderately 
large impact on the combined test scores for grade six (effect size == .39) and similarly large 
impacts on both reading and mathematics achievement test scores. 
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Going to a Private School 

Turning from the effect of a voucher offer to the impact of going to a private school for two 
years, column 7 of Tables 18-20 indicates no significant effects on test-score performance 
(combined) when all students are considered together.'*^’^® Nor were any significant effects 
observed on the test scores of Hispanic students.^' 

Here, too, the results for African Americans were noticeably different. For this group of 
students, the combined test scores of those going to a private school were, on average, 4.4 NPR 
points higher than similar students in the control group. The reading test scores of those going to 
a private school were, on average, 4.6 NPR points higher while mathematics scores were 4.2 
points higher. The estimate of the effect of going to a private school on the combined test-score 
performance was statistically significant. The reading score results were also statistically 
significant, but the mathematics score was not. 

An examination of the results by grade level reveals no statistically significant effects of 
going to a private school for African American students in grades three, four, and five two years 
after initiation of the program; however, significant effects were detected for students in grade 
six. Sixth-grade students going to a private school achieved, on average, a combined test score 
that was 10 NPR points higher than the score achieved by those who attended a public school 
(effect size = .48). Their mathematics score was 1 1 NPR points higher and their reading score 
8.9 points higher (effect sizes are .45 and .41). 

Putting the Achievement Results into Context 

Overall results after two years are similar to those observed after one year. In the first as in 
the second year of the voucher program, the effects of a voucher offer and of attending a private 
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school on the combined test scores of Hispanic students were not significant. Nor were any 
overall effects observed among all students after one year.^^ 

The results for African Americans were different, however. The effect of a voucher (offer or 
attending a private school) on the combined test-score performance of Afncan Americans after 
one year was 5-6 NPR points, a larger effect than the 3-4 point effect observed in year two. 
Given that the drop is not statistically significant, the most cautious interpretation of these results 
is that positive effects detected in year one carried over but did not increase in year two. It 
remains to be seen what impact on test scores will be observed after students have participated in 
the voucher program for three years. 

The effect size of the difference in combined test scores for African American students in 
NYC evaluation is .23, generally thought to be a moderately large effect of an education 
intervention. The difference between black and white test scores is roughly one standard 
deviation. This intervention, after two years, moderates a fifth of this difference. 

It remains to be seen whether results shown here will continue to be observed in subsequent 
years of the voucher experiment in New York City. 
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1999), Ch. 12. Similar evaluations of voucher initiatives in Washington, D. C. and Dayton, Ohio 
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To adjust for non-response, we used non-response adjusted sample weights (see appendix). 
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do not differ markedly from observed cases, the assumptions necessary for utilization of this 
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Voucher Programs: How do Applicants Differ from the Eligible Population?" Paper prepared for 
Program on Education Policy and Governance, Harvard University, forthcoming. Information is 
drawn from the Integrated Public Use Microdata Series data set of the U. S. Census, which has 
been created at the University of Minnesota. 

'^We computed the treatment differential (58 and 64 points) using two approaches. First, we 
compared the percent of treatments with two years of exposure to a private school (62 percent) 
with the percent of the control group that reported attending a private school for two years (4 
percent). Second, we compared the percent of the treatment group that attended a private school 
for one or more years (76 percent) with the percent of the control group with the same pattern of 
private school attendance (8 percent). 

Amy Stuart Wells, “African-American Students’ View of School Choice,” in Fuller and 
Elmore, eds.. Who Chooses? p. 47. 

Sandra Feldman, “Let’s Tell the Truth,” New York Times, November 2, 1997, p. 7 
(Advertisement). 
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Scholarship Program,” in Paul E. Peterson and Bryan C. Hassel., eds., Learning from School 
Choice (Washington, D. C.: Brookings, 1998), pp. 357-94. Paul E. Peterson, David Myers and 
William G. Howell, "An Evaluation of the Horizon Scholarship Program in the Edgewood 
Independent School District, San Antonio, Texas: The First Year," Occasional Paper, Program 
on Education Policy and Governance, Harvard University, Cambridge MA, October, 1999. 

Despite suspicions that families with more children would be less likely to be able to raise 
these supplemental funds and therefore more likely to decline the scholarship, a logit analysis 
revealed that no relationship exists between family size and the probability of declining a 
scholarship . 

Carnegie Foundation for the Advancement of Teaching, School Choice: A Special Report 
Princeton, New Jersey: Carnegie Foundation for the Advancement of Teaching, 1992), p. 13. 

Dan Murphy, F. Howard Nelson and Bella Rosenberg, "The Cleveland Voucher Program: 
Who Chooses? Who Gets Chosen? Who Pays?" (New York: American Federation of Teachers, 
1997), p. 10. 

Nicholas Lemann, "A False Panacea," Atlantic (January 1991), p. 104, as quoted in Abigail 
Themstrom, School Choice in Massachusetts (Boston: Pioneer Institute for Public Policy 
Research, 1991), p. 40. 

Seventy-five percent of those offered a scholarship made use of the scholarship; some non- 
users also reported finding a school they wanted. 

Murphy, Nelson, and Rosenberg, The Cleveland Voucher Program. 

Jay P. Greene, “Civic Values in public and Private Schools,” in Peterson and Hassel, eds. 
Learning from School Choice, pp. 83-106. 

Provided with large differences in school size and other characteristics of schools for members 
of the treatment and control groups, we plan in the future reports to assess the extent to which the 
differences may indicate impacts of vouchers on other outcomes. 

Peterson, Myers and Howell, 1998, table 5. 

Peterson, Myers, and Howell, 1998, Table 5. 

Peterson, Myers, and Howell, 1998, Table 5. 

Peterson, Myers, and Howell, 1998, table 6. 

Murphy, Nelson, and Rosenberg, The Cleveland Voucher Program. 

Peterson, Myers, and Howell, 1998, table 8. 

Peterson, Myers and Howell, 1998, table 8. 

For very similar first-year results, see Peterson, Myers, and Howell, 1998, table 9. 

Peterson, Myers and Howell, Table 9. 

Peterson, Myers and Howell, Table 10. 
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The impact of the program on religious practice after one year is reported in Peterson, Myers, 
and Howell, Table 1 1 . 

Brandi, Money and Good Intentions Are Not Enough. 

Peterson, Myers, and Howell, 1998, Table 13. 

Peterson, Myers and Howell, 1998, Tablel4. 

A summary of findings from earlier studies is available in Paul E. Peterson, “School Choice: A 
Report Card,” in Peterson and Hassel, Learning from School Choice, p. 18. Mark Schneider, 
Paul Teske, Melissa Marschall, and Christine Roch, “Tiebout, School Choice, Allocative and 
Productive Efficiency,” paper prepared for annual meetings of the American Political Science 
Association, 1998, finds higher levels of parental satisfaction within New York City public 
schools, when parents are given a choice of school. 

John F. Witte, “First Year Report; Milwaukee Parental Choice Program,” University of 
Wisconsin — Madison, Department of Political Science and Robert M. Lafayette Institute of 
Public Affairs, November 1991. 

Murphy, Nelson, and Rosenberg, The Cleveland Voucher Program: Who Chooses? Who Gets 
Chosen? Who Pays? 

Jay P. Greene, William G. Howell, and Paul E. Peterson, “Lessons from the Cleveland 
Scholarship Program,” in Peterson and Hassel, eds.. Learning from School Choice, pp. 376-80. 

Peterson, Myers, and Howell, 1998, table 16. 

These percentages may underestimate the actual rate of school mobility for both scholarship 
students and those in the control group. The families that did not attend questionnaire 
administration sessions probably were more likely to have moved, making it more difficult for 
evaluation staff to locate them. If so, the children in those families that could not be located 
would be more likely to have changed schools. In this regard, it is important to note that the 
response rate was less for the control group than for scholarship users. 

Peterson, Myers, and Howell, 1998, Table 16. 

Peterson, Myers, and Howell, 1998, Table 17. 

About 43 percent of those students identified as Hispanic are Puerto Ricans, about 40 percent 
are from the Dominican Republic, and the remaining 1 7 percent are identified as other. 

All effect sizes presented in the paper are in standard deviations. An effect size of .16 shows 
that a student in the voucher group with an average test score had a higher score than 56 percent 
of the students in the control group. 

To assess the impact of going to a private school, we used two different approaches. First, we 
computed the impact of ever attending a private school; the counterfactual is always attending a 
public school. Second, we computed the impact of attending a private school for two years; the 
counterfactual for this comparison is public school attendance and attendance in private schools 
for less than two years. Impact estimates using both approaches are similar and we present in the 



ever attending are displayed in Appendix E. 

As for the impact of a voucher offer, we find a significant impact on reading achievement in 
grade 6, but no overall impact. 

The approach for dealing with missing values in the constmction of the private school 
variable for the test score tables, and the tables reporting results from the parent and student 
surveys differed. We should note that missing values were only present for the control group; 
administrative records were used for the treatment group and these data were complete. Self- 
reports were used for the control group and there were missing data in some surveys. For the 
impacts based on responses from the parents and students, we constructed an indicator of 
whether a child attended a private school in year 2. If the control group parent did not respond to 
the private school item, we set the indicator to a missing value. For the test score tables we 
imputed values so that we could preserve sample points for the analyses that looked at impacts 
separately by grade level and by race/ethnicity; this approach stretched the data more thinly 
across groups. The following procedure was used to impute the values for attending a private 
school for two years in the test score analyses: (1) if missing in year 2 and not year 1, we 
assumed students attended the same kind of school both years and used the year 1 value; (2) if 
missing in year 1 and not year 2, we assumed student attended same kind of school both years 
and used the year 2 value, and (3) if school type was missing in year 1 and year 2, we assumed 
students attended the same kind of school as attended at baseline— because all students were 
attending a public school at baseline, this means we assumed they attended a public school in 
both follow-up years. 

The results presented in the first year report showed very small impacts for all students and 
were statistically significant when using one tailed-tests. This year, we have switched to using 
two-tailed tests based on both comments from reviewers and consideration on the range of 
questions policy makers and parents may wish to pose concerning whether there are no impacts 
or harmful effects associated with vouchers. There were small changes in the year one estimates 
because of improvements we made in the non-response adjusted weights. 
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RESPONSE RATES 
(Percentages^) 



Cl 

D 

? 

o 

o 

c 

o 

o 

0> 

c/5 



bo 



r- 

(U C 

"O .2 

c ^ 

D c/3 

< ^ 



a 

P 

I 

"o 

Uh 



bo 

.Si 

cs 

C< 



T 3 - 

0> c 
*o .2 
c ^ 

a> c« 

< c/5 



'4-.» 

C/3 


bo 


(U 


H 






cs 








C< 


T3 

0> 


c 


T3 


o 


C 


*c^ 


0> 


c/3 


•4^ 

< 


0> 

C/5 



On <N 
m VO 



m 

VO VO 



On 

VO VO 






Ov 



o 

VO 



Tj- oo 

r- r- 



On 

VO 



lO 00 (N 

r- r- r- 



oo VO 
r- oo 



r- 



m m m 

r- r- r- 



Ov Ov 

r- r- 



o 

o 



Ov 

r- 



I 

s 



< < :< 

Z Z Z 



< < :< 
z z z 



Tj- lO CN 

t — t — t — 



< :< :< 
Z Z Z 



:< < :< 
Z Z Z 



CN O 

00 00 00 



:< < :< 
Z Z Z 



^ ^ ^ 
z z z 



< < :< 
Z Z Z 



< < < 
Z Z Z 



VO 0\ CN 

VO VO 'O 



:< < :< 
Z Z Z 



< < :< 
z z z 



lO VO 

t — t — t — 



:< < :< 

Z Z Z 



< :< < 
Z Z Z 



(U 

> 

O 



O 


0 




0 


0 


0 




0 


0 


0 


O 


0 




0 


0 


0 




0 


0 


0 






















04 








04 








04 




D 








3 








3 




O 








0 








P 




Ui 

a 


D4 

3 






Ui 

a 


D4 

3 






u 

a 


D4 

3 


04 

IE 

c /3 


0 

Ui 

a 


Co 




04 

IE 

c /3 


0 

u 

a 


c 

Co 




IE 

c /3 


0 

u 

a 


Cd 


2 




Id 




2 






M 


"o 

u 


0 

•C 

0 

c /5 


a 

0 

U 


5 


Urn 

0 > 

> 

0 


0 

0 

c /5 


c 

0 

U 


u 

0 > 

> 

0 


0 

0 

CO 


C 

0 

U 



<L> 



<L> 



(D 

OJ) 

cd 



nC3 

C 

o 

o 

<L> 

C/3 

(L> 



•T3 

<D 

•w 

Cd 

a. 

o 

r 

cd 

a. 

o 



c 

(D 

•T3 



C/3 

(D 



E 

,cd 



cd 

O 

— 

a. 

E 

cd 

</3 

• 4 — • 

o 

• 4 —* 

c 

o 

nc3 

(D 

</3 

Cd 

X3 

(D 

• 4 — • 

cx 

E 

o 

o 

{/3 

<D 

Of) 

Cd 

c 

<D 

O 

Ui 

<D 

04 



LO 



CO 

LO 



o 

ERIC 



TABLE 2 



DEMOGRAPHIC CHARACTERISTICS FOR SCHOLARSHIP TAKERS AND DECLINERS” 

(Percentages) 





Takers 


Decliners 


Difference 




(1) 


(2) 


(3) 


Family income 








Less than $5,000 


25 


37 


-13*** 


$5,000-$ 10,999 


36 


40 


-4 


$11,000-S24,999 


36 


21 


15*** 


$25,000-$39,999 


4 


2 


2 


$40,000 or more 


0 


0 


0 


Total 


100 


100 




Average family income 


$10,419 


$7,732 


$2,686*** 


Family receiving following forms of government 








assistance: 








Welfare 


52 


67 


-15*** 


Social Security 


12 


10 


1 


Mother's employment status'^ 








Full time 


25 


16 


9*** 


Part time 


17 


15 


1 


Looking for work 


44 


51 


-7* 


Not looking 


13 


16 


-3 


Don’t Know 


1 


1 


0 


Total 


100 


100 




Percent of Mothers at Current Residence for 2 years or 


19 


24 


-6* 


Less 








Highest Level of education completed by Mother:^ 








Some high school 


19 


25 


-6* 


High school graduate or GED 


25 


27 


-2 

« * *** 


Some college 


45 


33 


11 


Graduated from a 4-year college 


8 


9 


-1 


More than a 4-year college degree 


3 


3 


-1 


Don’t know 


1 


3 


-2 


Total 


100 


100 




Mother's ethnicity:^^^ 








Black 


47 


45 


2 


White 


4 


9 


-6*** 


Puerto Rican 


17 


19 


-2 


Hispanic other than Puerto Rican 


26 


24 


2 


Other 


6 


3 


3* 


Total 


100 


100 
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TABLE 2 {continued) 





Takers 

(1) 


Decliners 

(2) 


Difference 

(3) 


Mother's religious affiliation:^ 


Baptist 


19 


23 


-4 


Other Protestant 


17 


15 


3 


Catholic 


54 


46 


8’ 


Other Religion 


5 


10 


-5” 


No Religion 


5 


6 


-1 


Total 


100 


100 




Percent of mothers US born 


59 


61 


-3 


Percent of households with English as main language 


80 


75 


4 


Percent of children receiving any special education 
services related to a disability or learning problem 


10 


14 


-4’ 


Baseline test scores (in national percentile rankings): 


Reading 


23 


23 


1 


Math 


17 


16 


1 


(N) 


506-860 


260-460 





Source: FAY30708.do 

*Statistically significant at .1. 

**Statistically significant at .05. 

’’^Statistically significant at .01 

^Significant at .01 using the chi-square. The chi-square test was used to test for differences in the distributions of categorical 
outcomes between takers and decliners. 

“ Takers are defined here as students in the treatment group who made use of the scholarship in year 2; decliners are students in 
the treatment group offered a scholarship but did not utilize it in year 2. 
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TABLE 3 



PERCENTAGE OF SCHOLARSHIP TAKERS THAT RESPONDED THAT THE FOLLOWING 
WAS ONE OF THE THREE MOST IMPORTANT IN CHOOSING A SCHOOL 





Percentage 


Std. Dev. 


That the child went to a neighborhood public school 


7 


26 


The school was the only choice available 


10 


30 


Academic quality 


59 


49 


Safety 


36 


48 


Religious instruction 


28 


45 


Convenient location 


20 


40 


The child’s friends 


0 


7 


The sports program 


0 


6 


The school facilities 


4 


19 


Discipline 


42 


49 


Teacher quality 


35 


48 


What is taught in class 


27 


44 


Class size 


21 


41 
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TABLE 4 



FAMILIES WHOSE CHILD ATTENDED PREFERRED SCHOOL 

(Percentage) 





Control Group^ 


Scholarship Offered^ 


Effect of Scholarship^ 


Attended preferred school: 

Reason for not attending 
preferred school: 


56 


79 


23*** 


Could not pay school cost 


31 


12 


-19-’ 


No space available 


4 


5 


1 


Applied too late 


1 


4 


3- 


Transportation problems 


3 




1 


School location 


2 


2 


0 


Child not given space 


4 


4 


0 


Child failed admission test 


2 


2 


0 


Not affiliated with church 


1 


1 


0 


Moved away from school 


1 


1 


0 


No reason given 


2 


1 


-1 



Weighted values reported. ^Effect of offer is statistically significant at .1 level, two-tailed test; Effect of offer 
is statistically significant at .05 level, two-tailed test; Effect significant at .01 level, two-tailed test. 



^ All applicants not offered a scholarship. Eight percent of the control group reported placing their child in 
private school; the remainder were in public school. 

^ Those who were offered a scholarship, whether or not they made use of it. 

^ Differences in outcomes between those offered a scholarship and those in control group. 

^ Parents could give more than one reason for not sending their child to a preferred school. 
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TABLE 5 



SIZE AND QUALITY OF SCHOOL FACILITIES 
(Percentages) 





Scholarship 

Offered^ 

(1) 


Control 
Groups 
(2) _ 


Scholarship 

Offer 

Impacf 

( 3 ) 


Scholarship 

User"* 

( 4 ) 


Control 

Group 

Compiler'^ 

(5) 


Switch to 
Private School*^ 
(6) 


Average school size 


403 


498 


-96*** 


385 


525 


-140*** 


Average class size 


25 


26 


-r* 


25 


27 


-2*** 


Percentage satisfied with 
school facilities 


30 


10 


20*** 


33 


5 


28*** 


Percentage with the 
following resources 














Special programs for non- 
English speakers 


52 


76 


-24*** 


44 


80 


-36*** 


Special programs for 
learning disabled 


63 


74 


-11*** 


58 


74 


-16*** 


Nurses’ office 


79 


94 


-15*** 


75 


96 


-21*** 


Child counselor 


77 


83 


-6* 


75 


83 


-8** 


Library 


89 


93 


-4 


88 


93 


-5* 


Cafeteria 


90 


96 


-6*** 


90 


98 


-8*** 


Special programs for 
advanced learners 


53 


58 


-5 


48 


55 


-7 


After-school program 


91 


90 


1 


92 


90 


2 


Gym 


91 


90 


1 


91 


89 


2 


Arts program 


81 


81 


0 


80 


79 


1 


Computer lab 


89 


84 


5 


90 


83 


7** 


Music program 


83 


77 


6* 


84 


75 


9*** 


Individual tutors 


58 


49 


9* 


57 


45 


12*** 


(N) 


889-1399 






889-1399 







Notes: 

“Those who were offered a scholarship, whether or not they made use of it. 

^ Those who were not offered a scholarship. 

‘^Estimated impact of being offered a scholarship. 

‘^Those who were offered a scholarship and identified by SCSF staff as having used their scholarship to attend a private 
school. 

^Those in the control group who would have used a scholarship had they been offered one as described in Appendix C. 
^Estimated impact of participation in the program, using a two-stage least squares model, as described in Appendix C. 
*Impact of offer is statistically significant at .1 level, two-tailed test; Impact of offer is statistically significant at .05 
level, two-tailed test; ***Impact significant at .01 level, two-tailed 
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TABLE 6 



ETHNIC AND RACIAL ISOLATION IN CLASSROOM 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to 
Private 
School 

(6) 


What percentage of 
students in child's class 
are minority? 














Less than half 


13 


12 


1 


13 


11 


2 


About half 


22 


21 


1 


25 


24 


1 


More than half 


35 


31 


4 


33 


27 


6 


Everyone 


30 


36 


-6 


30 


38 


8** 


Total 


100 


100 




101 


100 




(N) 


1402 






1402 ' 







See notes to Table 5. 
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TABLE? 



SPECIAL EDUCATION FACILITIES AND PROGRAMS 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 

Offer 

Impact 

(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to 
Private School 
(6) 


Children with physical 
disabilities 


4 


2 


2 


3 


1 


2 


Children with learning 
disabilities 


10 


10 


0 


9 


9 


0 


Enrolled in ESL course 


4 


5 


-1 


3 


5 


-2 


(N) 


1422-1425 






1422-1425 






Percentage who believe school 
doing 'very well ' at attending to 
these needs: 














Physical disabilities® 


35 


47 


-12 


33 


62 


-29 


Learning disabilities® 


30 


24 


6 


26 


16 


10 


ESL® 


31 


28 


3 


25 


19 


6 


(N) 


55-160 






55-160 







See notes to Table 5. 

“These figures are calculated as a percent of those parents with disabled or non-English speaking children, not as a percent 
of the entire population. 
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TABLES 



PARENT’S PERCEPTION OF SCHOOL CLIMATE 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to 
Private School 
(6) 


Parents report as serious problem: 














Fighting 


40 


66 


-26*** 


33 


70 


-37*** 


Tardiness 


39 


59 


-20*** 


33 


62 


-29*** 


Kids missing class 


34 


52 




29 


54 


-25*** 


Kids destroying property 


29 


42 




27 


45 


-18*** 


Cheating 


31 


40 


-9** 


29 


42 


-13*** 


Racial Conflict 


27 


37 


-10*** 


26 


41 


-15*** 


Parents report on school rules: 














School uniform 


84 


47 


37*** 


96 


43 


53*** 


Certain forms of dress forbidden 


87 


65 


22*** 


95 


64 


31*** 


Visitors must sign in at main office 


88 


94 


-6*** 


86 


95 


-9*** 


Hall passes required to leave class 


77 


86 


-9*** 


74 


' 86 


-12*** 


(N) 


1214-1397 






1214-1397 







See notes to Table 5. 
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TABLE 9 



STUDENT’S PERCEPTION OF SCHOOL CLIMATE 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 

Offer 

Impact 

(3) 


Scholarship 

User 

(4) 


Control 
Group 
Compiler 
(5) 


Switch to 
Private School 
(6) 


Student reports (percentages): 














Students are proud to attend this 
school 


59 


54 


5 


58 


52 


6 


Behavior rules are strict 


67 


62 


5^ 


70 


65 


5 


Students get along with teachers 


62 


50 


12'*' 


64 


49 


15'" 


Feel ‘put down’ by teachers 


21 


25 


-4 


20 


24 


-4 


Teachers ignore cheating 


17 


22 




16 


23 


-7" 


There is a lot of cheating in this 
school 


26 


34 




25 


36 


-11"' 


(N) 


1209-1274 






1144-211 






Student reports on number of close 
friends who:"^ 














Like school 


4.04 


4.06 


-0.02 


4.01 


4.04 


-0.03 


Get good grades 


4.68 


4.40 


0.28' 


4.68 


4.30 


0.38' 


Get into trouble with teachers 


2.26 


2.29 


-0.03 


2.33 


2.37 


-0.04 


Use bad languages 


1.89 


2.25 


-0.36" 


1.77 


2.25 


-0.48" 


Smoke cigarettes 


0.15 


0.12 


0.03 


0.12 


0.07 


0.05 


(N) 


1175-1195 






1175-1195 







See notes to Table 5. 

“The index is scored 0 if child reports no close friends at school, 1.5 for 1 to 2 friends, 3.5 for 3 to 4 friends, 5.5 for 5 to 6 
friends, and 7.5 for 7 or more friends. 
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TABLE 10 



HOMEWORK 

(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to 
Private School 
(6) 


Parents reports: 














Child has more than one hour of 
homework 


60 


44 


16*“ 


64 


41 


23*** 


Homework too easy 


7 


18 


-11“* 


4 


20 


-16*** 


(N)“ 


1410-1431 






1410-1431 






Student reports: 














Trouble keeping up with 
homework 


22 


26 


-4 


22 


28 


-6 


Time spent on homework on 
typical night (in minutes) 


50 


45 


6“ 


51 


44 


7** 


Teachers return homework 
always or most of time 


50 


54 


-4 


49 


55 


-6 


(Uf 


1275-1295 






1210-1295 







See notes to Table 5. 

“These values of (N) are drawn from the parent survey. 
'^These values of (N) are drawn from the student survey. 
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TABLE 1 1 



SCHOOL COMMUNICATION WITH PARENTS 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to 
Private School 
(6) 


Parents regularly informed about 


90 


83 


7*** 


94 


84 


10*** 


student grades 














Parents receive notes from teacher 


88 


79 




89 


76 


13*** 


Parents receive newsletter 


81 


65 


16*** 


85 


62 


23*** 


Notified of disruptive behavior 


88 


78 


10*** 


91 


77 


14*** 


Parents speak to classes about jobs 


37 


28 


9** 


36 


24 


12*** 


Parents participate in instruction 


62 


49 


13*** 


63 


44 


19*** 


Parent night 


91 


88 


3 


92 


88 


4 


Regular Parent-Teacher 


94 


91 


3 


95 


91 


4* 


Conferences 














Notified of disruptive behavior 


88 


78 


10*** 


91 


77 


14*** 


(N) 


1137-1401 






1137-1401 







See notes to Table 5. 
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TABLE 12 



RELIGIOUS PRACTICES 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

■ (4) 


Control 

Group 

Compiler 

(5) 


Switch to 
Private School 
(6) 


Student reports: 














Religious instruction outside school 


25 


16 


9“*“ 


27 


14 


13"‘ 


Attend religious services 


55 


35 


20**“ 


59 


32 


iT' 


Participate in church group 


47 


38 


9*** 


50 


40 


10" 


(N) 


1248-1273 






1184-1210 







See notes to Table 5. 
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TABLE 13 



PARENTAL INVOLVEMENT IN CHILD’S EDUCATION 





Scholarship 

User 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to Private 
School 
(6) 


Average number of times 
parents did the following'^ 














Helped child with homework 


11 


12 


0 


11 


12 


0 


Helped child with reading, math 


10 


10 


0 


10 


10 


0 


Talked with child about school 


13 


14 


0 


13 


14 


1 


Attend school activity with child 


5 


5 


0 


5 


5 


0 


Worked on school projects 


6 


5 


0 


6 


5 


1 


(N) 


1399-1424 






1399-1424 







See notes to Table 5. 

“ The index is scored 0 if a parent never did the activity, 3 for 1-5 times, 8 for 6-10 times, 13 for 11-15 times and 1 8 for 16 
or more times. 
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TABLE 14 



STUDENT ADJUSTMENT OF CHOICE SCHOOLS 





Scholarship 

User 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to Private 
School 
(6) 


Student reports: 














Number of close friends at 
school^ 


6 


5 


0 


6 


5 


0 


Percentage of students who 
feel “made fiin of by other 
students 


36 


38 


-2 


37 


39 


-2 


(N) 


1244-1254 






1178-1254 






Parent reports (Percentage): 














Child suspended in past year 


5 


4 


1 


3 


2 


1 


(N) 


1424 . 






1424 







See notes on Table 5. 

“The index is scored 0 if the child reports no close friends at school, 1 .5 for 1 to 2 friends, 3.5 for 3 to 4 friends, 5.5 for 5 to 
6 friends, and 7.5 for 7 or more friends. 
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TABLE 15 



PARENTAL AND STUDENT SATISFACTION WITH SCHOOL 
(Percent Very Satisfied) 





Scholarship 

User 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to Private 
School 
(6) 


Paren tal Sa tisf action : 














Observe religious traditions 


39 


9 


30*** 


47 


5 


42*** 


Class size 


30 


11 


19*** 


34 


7 


27*** 


Discipline 


43 


13 


30*** 


47 


5 


42*** 


Academic quality 


40 


13 


27*** 


44 


5 


39 


Student respect for teachers 


45 


17 


28*** 


50 


11 


39*** 


Parental support 


33 


11 


22*** 


37 


6 


31*** 


Teaching values 


37 


14 


23*** 


40 


7 


33*** 


What taught in school 


42 


14 


28*** 


47 


7 


40*** 


School safety 


47 


16 


31*** 


52 


9 


43*** 


Teaching 


47 


18 


29*** 


51 


10 


41*** 


Teacher-parent communication 


43 


22 


21*** 


49 


19 


30*** 


Clarity school goals 


33 


12 


21*** 


36 


6 


30*** 


Staff teamwork 


30 


12 


18*** 


32 


6 


26*** 


Sports program 


20 


8 


12*** 


23 


6 


17*** 


School facility 


30 


10 


20*** 


33 


5 


28*** 


Parental involvement 


33 


17 


16*** 


35 


12 


23*** 


Location 


50 


32 


18*** 


53 


28 


25*** 


Gave school an ‘A’ 


32 


11 


21*** 


38 


9 


29*** 


(N) 


1354-1436 






1354-1436 






Student reports: 














Gave school an ‘A’ 


52 


46 


6** 


53 


45 


8** 


Gave school ‘D’, ‘F’ 


4 


9 


-5*** 


4 


11 


-7*** 


(N) 


1359 






1359 







See notes to Table 5. 
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TABLE 16 



STUDENTS CHANGING SCHOOL DURING SCHOOL YEAR 
(Percentages) 





Scholarship 

User 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 

(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to Private 
School 
(6) 


Attended same school for 
entire school year 


95 


94 


1 - 


97 


95 


2 


Reasons why did not attend 
same school for entire year: 














Moved away 


2 


2 


0 


1 


1 


0 


Quality of school 


1 


2 


-1 


0 


2 


-2 


School too expensive 


1 


1 


0 


1 


0 


1 


Suspended/expelled 


0 


0 


0 


0 


0 


0 


Preferred public school 


0 


0 


0 


0 


0 


0 


Inconvenient location 


0 


1 


-T’ 


0 


1 


-r 


Preferred private school 


1 


1 


0 


0 


0 


0 


(N) 


1436 






1436 







^See notes on Table 5. 
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TABLE 17 



SCHOOL MATRICULATION PLANS FOR NEXT SCHOOL YEAR 

(Percentages) 





Scholarship 

User 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group 

Compiler 

(5) 


Switch to Private 
School 
(6) 


Child will attend same school 
next year 


71 


56 


15 


81 


60 


21 


Reasons why student not attend 
same school for next year: 














Quality of school 


6 


8 


-2 


4 


7 


-3 


Moving 


5 


5 


0 


4 


4 


0 


Graduating 


5 


16 


-ir** 


2 


17 


-15*** 


Preferred private school 


2 


2 


0 


2 


2 


0 


Inconvenient location 


3 


1 


2** 


3 


1 


2** 


School too expensive 


3 


2 


1 


2 


0 


2 


Children in same school 


1 


2 


-r 


1 


3 


-2* 


Asked not to return 


0 


0 


0 


0 


-1 


r* 


Preferred public school 


1 


1 


0 


1 


1 


0 


(N) 


1429 






1429 







See notes on Table 5. 
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YEAR TWO COMPOSITE TEST SCORE IMPACTS AND IMPACT 
DIFFERENCES BETWEEN YEAR ONE AND TWO 
(Percentile) 
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See notes to Table 5. 

‘^Values of (N) are the range in the unweighted number of people who responded to specific items. 

Appendix C describes how between year differences were tested. ,,, 

‘impact of offer is statistically significant at .1 level, two tailed test; “impact of offer is statistically significant at .05 level, two-tailed test; Impact significant at .01 level, 
two-tailed test. 
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YEAR TWO MATH TEST SCORE IMPACTS AND IMPACT 
DIFFERENCES BETWEEN YEAR ONE AND TWO 
(Percentile) 
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See notes to Tables 5 and 18. 



TABLE 21 



YEAR ONE READING TEST SCORE IMPACTS 



Control Switch to 

Scholarship Control Scholarship Scholarship Group Private 

Offered Group Offer Impact (N) User Complier School (N) 

(IJ (2) (3]_ (4} (5) (7) (8}_ 

All students: 



Overall - Average 


25.79 


24.75 


1.04 


Grade 3 


25.90 


23.45 


2.45 


Grade 4 


22.08 


24.87 


-2.79 


Grade 5 


28.93 


28.68 


0.24 


Grade 6 


26.77 


22.25 


4.52** 



African American students: 



Overall - Average 


25.62 


22.10 


3.53** 


Grade 3 


28.22 


22.40 


5.82 


Grade 4 


21.74 


25.47 


-3.73 


Grade 5 


24.81 


22.43 


2.38 


Grade 6 


28.43 


19.47 


8.97*** 



Hispanic students: 



Overall - Average 


23.94 


24.64 


-0.70 


Grade 3 


23.67 


21.51 


2.16 


Grade 4 


19.58 


22.51 


-2.93 


Grade 5 


28.44 


28.34 


0.10 


Grade 6 


24.51 


22.45 


2.06 



1456 


25.38 


23.96 


1.42 


1450 


371 


26.24 


22.79 


3.45 


370 


396 


21.57 


25.50 


-3.93 


393 


395 


28.49 


28.24 


0.25 


394 


294 


25.75 


19.82 


5.93** 


293 



624 


26.02 


21.46 


4.56** 


623 


153 


28.18 


20.40 


7.78 


153 


180 


21.28 


26.08 


-4.80 


179 


167 


26.39 


23.16 


3.23 


167 


124 


28.93 


28.64 


10.29** 


124 



709 


22.54 


23.55 


-1.01 


704 


188 


22.40 


19.13 


3.27 


187 


184 


17.99 


22.37 


-4.38 


182 


196 


27.57 


27.49 


0.08 


195 


141 


22.12 


19.23 


2.89 


140 



See notes to Tables 5 and 1 8. 
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TABLE 22 



YEAR ONE MATH TEST SCORE IMPACTS 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


(N) 

(4) 


Scholarship 

User 

(5) 


Control 

Group 

Compiler 

(6) 


Switch to 
Private School 

0 


(N) 




All students: 


















Overall - Average 


24.73 


23.31 


1.42 


1456 


23.27 


21.36 


1.90 


1450 


Grade 3 


20.05 


16.78 


3.27 


371 


20.21 


15.46 


4.75 


370 


Grade 4 


20.36 


22.61 


-2.25 


396 


18.28 


21.26 


-2.97 


393 


Grade 5 


31.15 


27.28 


3.87 


395 


29.93 


24.73 


5.20 


394 


Grade 6 


29.29 


24.40 


4.89* 


294 


26.34 


19.99 


6.35* 


293 


African-American students: 
















Overall - Average 


23.43 


18.01 


5.42*** 


624 


22.68 


15.67 


7.01*** 


623 


Grade 3 


20.54 


11.65 


8.89*** 


153 


21.14 


9.25 


11.89*** 


153 


Grade 4 


19.24 


19.20 


0.05 


180 


17.02 


17.09 


-0.07 


179 


Grade 5 


26.87 


21.52 


5.35 


167 


27.07 


19.82 


7.25 


167 


Grade 6 


29.38 


20.39 


8.99** 


124 


27.61 


17.28 


10.32** 


124 


Hispanic students: 


















Overall - Average 


24.19 


24.82 


-0.63 


709 


22.58 


23.60 


-1.02 


704 


Grade 3 


19.74 


18.06 


1.68 


188 


19.68 


17.03 


2.64 


187 


Grade 4 


19.03 


20.18 


-1.15 


184 


16.97 


18.69 


-1.72 


182 


Grade 5 


30.91 


26.78 


4.13 


196 


29.79 


24.62 


5.17 


195 


Grade 6 


28.98 


32.59 


-3.60 


141 


24.64 


29.55 


-4.91 


140 



See notes to Tables 5 and 1 8. 
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TABLE A-1 



BASELINE CHARACTERISTICS FOR TREATMENT AND CONTROL GROUPS 

(Mean Values Reported) 



Variable 


Control 

group 


Treatment 

group 


Difference 


t-stat 


Sig- 


Grade of Student ('96-97) 


Kindergarten 


0.17 


0.16 


0.01 


0.62 


_ 


First 


0.18 


0.20 


-0.0 1 


-0.76 





Second 


0.22 


0.21 


0.01 


0.42 


_ 


Third 


0.23 


0.23 


0.00 


0.25 





Fourth 


0.20 


0.20 


-0.01 


-0.53 


- 


Years student attended this school 


2.47 


2.39 


0.08 


1.37 


- 


Satisfaction with aspects of current school 


Location 


2.94 


3.00 


-0.06 


-1.44 





School Safety 


2.75 


2.79 


-0.04 


-1.01 


_ 


Teaching 


2.70 


2.66 


0.04 


1.07 


_ 


How much school involves parents 


2.72 


2.72 


0.01 


0.24 





Class sizes 


2.32 


2.34 


-0.01 


-0.34 


_ 


School Facilities 


2.63 


2.61 


0.02 


0.61 


_ 


Student respect of teachers 


2.86 


2.89 


-0.03 


-0.77 





Parent-teacher communication 


2.81 


2.81 


0.01 


0.14 





Observation of religious traditions 


2.31 


2.24 


0.07 


1.71 


- 


Student in gifted classes 


0.12 


0.12 


0.00 


0.21 


- 


Student received help for disability 


0.10 


0.12 


-0.02 


-1.28 


- 


Mother's educational expectations for child 


16.67 


16.76 


-0.09 


-1.07 




(10=some HS, 12=HS grad, 14=some college, 
16=college grad, 18=more than college) 

Education level of mother or female guardian 


Some high school (did not graduate) 


0.22 


0.21 


0.01 


0.44 


_ 


High school graduate or GED 


0.27 


0.25 


0.02 


0.78 


_ 


Some college 


0.40 


0.41 


-0.01 


-0.34 





Graduated from 4-year college 


0.07 


0.08 


-0.01 


-0.91 





More than 4-year college degree 


0.03 


0.03 


0.00 


0.33 





Don't know 


0.01 


0.02 


-0.01 


-1.45 


- 


Race/ethnicity of mother/female guardian 


White 


0.05 


0.06 


-0.01 


-1.08 




Black 


0.44 


0.47 


-0.02 


-0.91 




Puerto Rican 


0.21 


0.18 


0.03 


1.48 




Hispanic other than Puerto Rican 


0.27 


0.25 


0.02 


0.83 


_ 


Other 


0.03 


0.05 


-0.01 


-1.28 
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TABLE A-1 (continued) 





Control 


Treatment 






Variable 


group 


group Difference 


t-stat 


Sig- 



Birth place of mother/female guardian 



Bom in United States 


0.61 


0.59 


0.02 


0.84 




Bom in Puerto Rico 


0.09 


0.08 


0.01 


0.80 




Bom outside U.S. and Puerto Rico 


0.30 


0.33 


-0.03 


-1.37 


— 


Length of residence of mother in months 


36.09 


35.89 


0.20 


0.36 


- 


Job status of mother /female guardian 


Full-time job 


0.22 


0.22 


0.00 


0.00 





Part-time job 


0.15 


0.16 


-0.02 


-0.96 





Not working now but looking for work 


0.46 


0.47 


-0.01 


-0.32 





Not working and not looking for work 


0.15 


0.14 


0.02 


0.91 


— 


Don't know 


0.02 


0.01 


0.01 


1.46 


- 


Religious affiliation of female guardian 
Catholic 


0.52 


0.51 


0.01 


6.56 




Religion other than Catholic 


0.42 


0.44 


-0.02 


-0.91 





None 


0.06 


0.05 


0.01 


0.76 


- 


Number of children in home 


2.38 


2.34 


0.05 


0.86 




In child's home (percent saying yes): 


A daily newspaper 


0.84 


0.85 


-0.01 


-0.57 


_ 


An encyclopedia 


0.71 


0.71 


-0.01 


-0.24 





A dictionary 


0.98 


0.97 


0.01 


0.83 





More than 50 books 


0.85 


0.85 


0.00 


-0.20 


- 


Member of household receiving assistance: 


Food stamps 


0.67 


0.66 


0.01 


0.62 





Welfare 


0.58 


0.57 


0.01 


0.36 




Social Security 


0.11 


0.11 


0.00 


-0.01 


_ 


Medicaid 


0.66 


0.63 


0.04 


1.63 




Supplemental Security Income 


0.14 


0.13 


0.00 


0.06 


- 


Family income 


9450.23 


9466.92 


-16.69 


-0.05 


- 


Reading Achievement Scores 


Overall 


24.52 


22.88 


1.64 


1.58 




1** grade cohort 


27.29 


22.02 


5.27 


2.39 


♦ ♦ 


2"*^ grade cohort 


25.05 


25.10 


-.05 


-.02 





3"^ grade cohort 


21.03 


20.35 


.67 


.40 


_ 


4*^ grade cohort 


24.80 


24.40 


.40 


.18 


- 


Math Achievement Scores 


Overall 


17.12 


17.06 


0.05 


0.06 




grade cohort 


11.88 


10.83 


1.05 


.74 




2"^^ grade cohort 


19.14 


19.51 


.37 


.23 




3"^ grade cohort 


17.16 


16.66 


.51 


.26 


_ 


4*^ grade cohort 


21.32 


22.79 


1.47 


.57 


- 


English spoken at home 


0.75 


0.78 


-0.03 


-1.45 






Source: fay30300.xls,fay307l3.do 
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TABLE A-2 



DIFFERENCES IN BASELINE CHARACTERISTICS FOR RESPONDENTS AND 
NON-RESPONDENTS IN SECOND FOLLOW-UP: TREATMENT GROUP 
(Mean Values Reported) 



Non- 



Variable 


Respondents 


Respondent 


Difference 


t-stat 


S.g. 


Grade of Student (V6- '97) 


Kindergarten 


0.16 


0.12 


0.05 


1.86 


- 


First 


0.18 


0.21 


-0.03 


-1.10 


— 


Second 


0.21 


0.21 


0.00 


-0.07 


_ 


Third 


0.23 


0.26 


-0.02 


-0.71 


_ 


Fourth 


0.21 


0.21 


0.01 


0.20 


- 


Years student attended this school 


2.46 


2.37 


0.09 


0.97 


- 


Satisfaction with aspects of current school 


Location 


3.03 


2.93 


0.10 


1.57 


_ 


School Safety 


2.81 


2.62 


0.19 


2.90 


*** 


Teaching 


2.66 


2.58 


0.08 


1.34 


- 


How much school involves parents 


2.72 


2.57 


0.15 


2.54 




Class sizes 


2.35 


2.26 


0.08 


1.27 




School Facilities 


2.61 


2.52 


0.09 


1.56 


_ 


Student respect of teachers 


2.91 


2.71 


0.19 


2.98 


*** 


Parent-teacher communication 


2.84 


2.7 


0.14 


2.21 


** 


Observation of religious traditions 


2.25 


2.25 


0.00 


0.00 




Student in gifted classes 


0.11 


0.13 


-0.02 


-1.00 


- 


Student received help for disability 


0.11 


0.15 


-0.04 


-1.58 


- 


Mother's educational expectations for child 


16.77 


16.55 


0.22 


1.63 





(10=some HS, 12=HS grad, 14=some 
college, 16=college grad, 18=more than 
college) 

Education level of mother or female guardian 


Some high school (did not graduate) 


0.24 


0.21 


0.03 


1.05 




High school graduate or GED 


0.24 


0.27 


-0.04 


-1.05 




Some college 


0.40 


0.40 


0.00 


0.01 


— 


Graduated from 4-year college 


0.07 


0.07 


0.00 


0.22 


— 


More than 4-year college degree 


0.03 


0.03 


-0.01 


-0.57 


— 


Don't know 


0.02 


0.01 


0.01 


0.60 


- 


Race/ethnicity of mother/female guardian 


White 


0.03 


0.08 


-0.05 


-2.65 


*** 


Black 


0.45 


0.53 


-0.09 


-2.30 




Puerto Rican 


0.18 


0.18 


0.00 


-0.01 





Hispanic other than Puerto Rican 


0.29 


0.18 


0.11 


3.76 


♦ ♦ ♦ 


Other 


0.05 


0.03 


0.02 


1.78 
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TABLE A-2 {continued) 



Variable 


Respondents 


Non- 

Respondents 


Difference 


t-stat 


Sig. 


Birth place of mother/female guardian 


Bom in United States 


0.57 


0.67 


-0.10 


-2.65 




Bom in Puerto Rico 


0.08 


0.08 


0.00 


-0.02 





Bom outside U.S. and Puerto Rico 


0.35 


0.25 


0.10 


2.80 


*** 


Length of residence of mother in months 


36.73 


34.27 


2.45 


2.53 


** 


Job status of mother /female guardian 


Full-time job 


0.22 


0.19 


0.03 


1.02 





Part-time job 


0.16 


0.15 


0.00 


0.16 




Not working now but looking for work 


0.44 


0.52 


-0.08 


-2.13 


** 


Not working and not looking for work 


0.16 


0.12 


0.04 


1.71 


♦ 


Don't know 


0.01 


0.01 


0.00 


0.19 


- 


Religious affiliation of female guardian 


Catholic 


0.53 


0.43 


0.10 


2.63 




Religion other than Catholic 


0.43 


0.49 


-0.06 


-1.57 


- 


None 


0.04 


0.08 


-0.04 


-2.08 




Number of children in home 


2.43 


2.42 


0.01 


0.08 


- 


In child's home (percent saying yes): 


A daily newspaper 


1.16 


1.15 


0.01 


0.21 


_ 


An encyclopedia 


1.30 


1.29 


0.01 


0.25 


_ 


A dictionary 


1.03 


1.03 


0.00 


-0.12 


_ 


More than 50 books 


1.15 


1.17 


-0.01 


-0.48 


- 


Member of household receiving assistance: 


Food stamps 


0.65 


0.72 


-0.07 


-2.14 


** 


Welfare 


0.55 


0.64 


-0.09 


-2.52 


** 


Social Security 


0.12 


0.11 


0.01 


0.37 





Medicaid 


0.63 


0.68 


-0.05 


-1.47 




Supplemental Security Income 


0.15 


0.15 


0.00 


-0.01 


- 


Family income 


9875 


8451.54 


1423.46 


2.91 


*** 


Reading Achievement Scores 


Overall 


22.34 


21.92 


0.42 


0.23 




1 grade cohort 


22.80 


22.80 


0.00 


0.00 




2"*^ grade cohort 


23.44 


23.27 


0.17 


0.04 


_ 


3^*^ grade cohort 


19.59 


19.07 


0.52 


0.18 


_ 


4*^ grade cohort 


24.04 


23.26 


0.78 


0.21 


- 


Math Achievement Scores 


Overall 


17.29 


16.03 


1.26 


0.72 




V' grade cohort 


9.68 


10.17 


-0.50 


-0.18 




2"*^ grade cohort 


20.54 


17.25 


3.29 


1.02 


_ 


3*^*^ grade cohort 


17.41 


13.77 


3.63 


1.30 


_ 


4*^ grade cohort 


19.81 


23.50 


-3.69 


-0.78 


- 


English spoken at home 


0.75 


0.84 


-0.10 


-3.26 


*** 



Source: fay30903.xls,fay30903.do 
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TABLE A-3 



DIFFERENCES IN BASELINE CHARACTERISTICS FOR RESPONDENTS AND 
NON-RESPONDENTS IN SECOND FOLLOW-UP: CONTROL GROUP 
(Mean Values Reported) 



Non- 



Variable 


Respondents 


Respondents 


Difference 


t-stat 


Sig- 


Grade of Student (*96-97) 


Kindergarten 


0.16 


0.16 


0.01 


0.27 


- 


First 


0.19 


0.13 


0.06 


2.20 




Second 


0.21 


0.23 


-0.02 


-0.74 


— 


Third 


0.23 


0.25 


-0.02 


-0.48 


_ 


Fourth 


0.19 


0.22 


-0.03 


-0.87 


- 


Years student attended this school 


2.47 


2.55 


-0.08 


-0.88 


- 


Satisfaction with aspects of current school 


Location 


2.92 


2.86 


0.06 


0.85 


— 


School Safety 


2.74 


2.62 


0.12 


1.85 




Teaching 


2.70 


2.59 


^ 0.11 


1.76 




How much school involves parents 


2.75 


2.60 


0.15 


2.56 


♦ ♦♦ 


Class sizes 


2.35 


2.22 


0.13 


1.93 


* 


School Facilities 


2.62 


2.55 


0.07 


1.07 




Student respect of teachers 


2.86 


2.74 


0.12 


1.90 


* 


Parent-teacher communication 


2.82 


2.71 


0.11 


1.72 


* 


Observation of religious traditions 


2.32 


2.21 


0.11 


1.51 


- 


Student in gifted classes 


0.11 


0.12 


-0.01 


-0.56 


- 


Student received help for disability 


0.12 


0.08 


0.04 


1.80 


* 


Mother's educational expectations for child 


16.66 


16.73 


-0.07 


-0.55 


_ 



(10=some HS, 12=HS grad, 14=some college, 
1 6=college grad, 1 8=more than college) 



Education level of mother or female guardian 



Some high school (did not graduate) 


0.23 


0.23 


0.00 


-0.08 


_ 


High school graduate or GED 


0.28 


0.26 


0.02 


0.68 




Some college 


0.39 


0.39 


0.01 


0.17 


— 


Graduated from 4-year college 


0.06 


0.09 


-0.03 


-1.53 


— 


More than 4-year college degree 


0.03 


0.02 


0.01 


0.85 


— 


Don't know 


0.01 


0.02 


-0.01 


-0.58 


- 


Race/ethnicity of mother/female guardian 


White 


0.03 


0.03 


0.00 


0.38 


_ 


Black 


0.42 


0.49 


-0.08 


-2.07 




Puerto Rican 


0.22 


0.21 


0.01 


0.28 


_ 


Hispanic other than Puerto Rican 


0.31 


0.21 


0.09 


3.01 


*** 


Other 


0.02 


0.05 


-0.03 


-2.07 


** 
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TABLE A-3 {continued) 





Non- 






Variable 


Respondents Respondents Difference 


t-stat 


Sig- 



Birth place of mother/female guardian 



Bom in United States 


0.60 


0.66 


-0.06 


-1.57 





Bom in Puerto Rico 


0.10 


0.07 


0.03 


1.49 


_ 


Bom outside U.S. and Puerto Rico 


0.30 


0.27 


0.03 


0.74 


- 


Length of residence of mother in months 


36.67 


36.03 


0.64 


0.74 


- 


Job status of mother/female guardian 


Full-time job 


0.21 


0.22 


-0.02 


-0.52 


_ 


Part-time job 


0.14 


0.16 


-0.02 


-0.58 


_ 


Not working now but looking for work 


0.46 


0.45 


0.02 


0.44 


_ 


Not working and not looking for work 


0.17 


0.14 


0.03 


1.04 


_ 


Don't know 


0.02 


0.03 


-0.01 


-0.97 


- 


Religious affiliation of female guardian 


Catholic 


0.57 


0.48 


0.09 


2.56 


** 


Religion other than Catholic 


0.38 


0.47 


-0.09 


-2.50 


** 


None 


0.05 


0.05 


0.00 


-0.16 


- 


Number of children in home 


2.41 


2.51 


-0.11 


-1.06 


- 


In child*s home (percent saying yes): 


A daily newspaper 


1.16 


1.15 


0.01 


0.37 


_ 


An encyclopedia 


1.30 


1.29 


0.01 


0.22 


_ 


A dictionary 


1.02 


1.02 


0.00 


-0.44 


_ 


More than 50 books 


1.15 


1.15 


0.00 


0.10 


- 


Member of household receiving assistance: 


Food stamps 


0.67 


0.71 


-0.04 


-1.13 


_ 


Welfare 


0.58 


0.63 


-0.05 


-1.30 





Social Security 


0.12 


0.09 


0.03 


1.28 





Medicaid 


0.67 


0.67 


0.01 


0.19 


_ 


Supplemental Security Income 


0.16 


0.09 


0.07 


2.63 


*** 


Family income 


9303.63 


9330.68 


-27.05 


-0.05 


- 


Reading Achievement Scores 


Overall 


23.83 


23.83 


0.00 


0.00 


_ 


1^^ grade cohort 


26.25 


33.30 


-7.05 


-1.69 


* 


2"*^ grade cohort 


25.67 


20.62 


5.05 


1.34 


_ 


3^*^ grade cohort 


20.99 


19.50 


1.49 


0.58 





4*^ grade cohort 


22.98 


25.56 


-2.58 


-0.74 


- 


Math Achievement Scores 


Overall 


17.37 


17.02 


0.35 


0.22 


_ 


grade cohort 


10.47 


17.51 


-7.04 


-2.43 


** 


2”*^ grade cohort 


19.95 


16.22 


3.72 


1.28 





3^*^ grade cohort 


17.87 


13.35 


4.52 


1.57 





4*^ grade cohort 


20.76 


21.53 


-0.76 


-0.19 


- 


English spoken at home 


0.73 


0.81 


-0.07 


-2.42 


** 



Source: fay30903.xls,fay30903.do 
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Appendix B 



Adjusting Sample Weights for Non-Response 
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Adjusting Sample Weights for Non-Response 

Families within the sample had different probabilities of being offered a scholarship. To 
reflect these differences in the probability of selection and to reflect the composition of the 
population of eligible applicants, we weight the sample data. The weights were constructed by 
taking the inverse of the probability of being selected for a scholarship. Weights for scholarship 
families were multiplied by .217 and weights for control group families were multiplied by .783 
to reflect the ratio of treatment to control group families in the initial pool of eligible applicants. 
All weights were divided by 2 to sum to the size of the population we are trying to represent, not 
twice the population. In this sample, the average weight was about 4.2. A family with a weight 
of 4.2 stands in for 3.2 other families in the pool of applicants as well as itself The weights, 
which were adjusted for the same family applying multiple times, range in size from about .5 to 
22 . 



About 18 percent of all families in the first year and 26 percent of families in the second 
year did not complete a survey. To adjust for this non-response, we computed the probability of 
responding based on a logit model. The independent variables in the logit model included family 
characteristics such as race/ethnicity, number of siblings, language spoken at home, mother’s 
education, and family income, and other variables used to stratify the sample when we collected 
the baseline data. After computing the predicted probability of responding, we adjusted the 
baseline weight as follows: 

Wi=l/[fi*pi*pri] 

where fi includes the adjustment factors used for deriving the baseline weight, p^ is the 
probability of being selected for a scholarship (control group), pr^ is the probability of 
responding for each follow-up survey, and Wi is the new weight variable. Families that did not 
respond to the follow-up survey were assigned a weight of zero. 

For the second year student data, we found that 66 percent of the students responded to the 
survey and that we had test scores for 66 percent. During the previous round of data collection, 
75 percent of students completed the survey and 78 percent took the achievement test. To adjust 
the weights for the student level data, we followed the same procedures that were used for the 
parent data. 



' The control group was reduced to 1,293 students from the initial eligible population of 
5,658 and the treatment group was reduced from 1,558 to 1,374 students. The weights for the 
reduced sample were re-scaled to sum to the initial eligible population. 

^ The adjustment factors are as follows: 1) five discrete points at which families applied for 
scholarships; 2) whether a child attended a public school with below average achievement; 3) the 
number of eligible children within the family. 
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Analytic Approach 



For some analysts and program operators, the important policy question is as follows: what 
happens when a voucher or scholarship program is put into effect? But other analysts also want 
an answer to a second question: what is the impact of attending a private school? Angrist, 
Imbens, and Rubin (1996) make some interesting distinctions between the two estimators used to 
address these two different questions. We refer to the first estimator as the intended to treat 
(ITT) estimator and the second as the complier average causal effect estimator (CAGE). The 
most important issues concerning the ITT estimator, which compares all children randomly 
assigned to the scholarship group with all children randomly assigned to the control group, are 
that among the children who are assigned to the two groups are children who are induced by the 
offer of a scholarship to attend a private school, children who would have made the decision to 
attend a private school regardless of the scholarship offer, and children who would never attend a 
private school. The CAGE estimator provides an estimate of the impact of the scholarship for 
only those who were or would have been induced by the offer of a scholarship to attend private 
school. Ghildren who would have attended a private school regardless of the offer of a 
scholarship and those who would have opted to not attend irrespective of the scholarship do not 
play a direct role in the estimated impact with the GAGE estimator. 



Gomputing Impacts of Being Offered a Scholarship (ITT) 

To compute the impact of being offered a scholarship we use a simple statistical model that 
includes as independent variables an indicator for treatment status (offered a scholarship or in the 
control group) and a set of indicators that show the stratum from which a family was selected. 
The strata are based on (1) five discrete points at which families applied for scholarships, (2) 
whether a child attended a public school with below average achievement, and (3) the number of 
eligible children within the family. When computing the impact on student achievement test 
scores, we also included student baseline reading and math achievement. The basic form of the 
model is: 



yu ==/5o + /5,7’, + /32X+ei, 
yv"" Pi '^P^Ti + Ps Xi + £2i 



where 7) equals 1 if we offered a family a scholarship and 0 otherwise (families were 
randomly selected for the scholarship and control groups); X, is a vector that includes indicator 
variables for each of the strata used in the random selection of scholarship families and baseline 
test scores when computing impacts on achievement. The outcomes of interest areyi, and yii. 
The former is the first year test score and the latter is the second year test score. The random 
error terms, ei, and £ 2 ,, capture the effects of unobserved factors that influence the outcomes; and 
the P ‘s are parameters or vectors of parameters to be estimated. The parameters of most interest 

are P, and (5^ because they show the impact of being offered a scholarship on the outcome for 
year 1 and year 2, respectively. We estimate the model parameters by using ordinary least 
squares for both categorical and continuous outcomes. 
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Computing theComplier Average Causal Effect (CACE) 

A simple comparison of an outcome for families in the scholarship group (those whom we 
offered a scholarship) and the control group shows the impact of being offered a scholarship, 
regardless of whether a family sent their child to a private/parochial school or not. 

To compute the CACE estimator for the impact of attending a private school, we need to 
estimate a statistical model that focuses on (1) the relationship between being offered a 
scholarship and attending private school and (2) the relationship between attending private 
school and family and student outcomes. These relationships can be expressed as: 

P\i=aQ + a\Ti + a2Xi+£p\i 

J'l,= ^0+ P,Xi+eyu 

P 2i ~ CC2'^(X4Ti'^OlsXi'^£p2i 
y 2 i~ P "i ^ P 4 P i 2 P 5 X i ~^ £ v 2 i 



where Ti equals 1 if we offered a family a scholarship and 0 otherwise (families were 
randomly selected for the scholarship and control groups); Xi is a vector that includes indicator 
variables for each of the stratum used in the random selection of scholarship families and 
baseline test scores when computing impacts on achievement; P/, if attended a private school in 
year one and P 21 equal 1 if a family attended a private school in year one and year two, and 0 
otherwise; yu and >’ 2 / are the outcomes of interest; Epi,, £, 1 /, Epu and £, 2 / are random error terms that 
capture the effects of unobserved factors that influence both private school attendance and the 
outcome; and a‘s and (3 ‘s are parameters or vectors of parameters to be estimated. We allow 
for across equation error correlations only within time periods. The parameters of most interest 
are /3, and because they show the impact of attending a private school on the outcome. 

We estimate the model parameters by using the instrumental variables estimator. This 
technique allows us to compute asymptotically unbiased and efficient estimates of the 
parameters; which can be interpreted as the causal impact for compliers (students that were 
induced to attend a private school by the scholarship offer) using the framework developed by 



‘ For analyses of the parent and student survey data, we focused in attendance at a private 
school in year two only. In this case, P ,2 == 1 if attended a private school in year 2, and 0 
otherwise. 

^ As already described in the report, we used two definitions of private school attendance 
when analyzing the test scores: (1) ever attend a private school, and (2) attended a private school 
for two years. We discuss the results of attending for two years in the report (Appendix E shows 
results for ever attending, which are quite close to the estimates of attending for two years). The 
impacts for ever attending are implemented by making a small adjustment to the analytic models 
described in this appendix. 

O 
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Angrist, Imbens, and Rubin, 1996. To implement the instrumental variables estimator, we use 
the two-stage least squares procedure. 

In our tables we present (1) the impact of attending private school, (2) the average of each 
outcome for families or students in the scholarship group that attended private school (complied), 
and (3) the average of outcomes for families or students in the control group who would have 
attended a private school if offered a scholarship. The first quantity is obtained from the 
statistical model described previously. The average for compilers in the scholarship group is 
computed by adding the impact of attending private school to the average for members of the 
control group who would have complied. To compute the last quantity, we can use an alternative 
expression for computing the impacts of private school attendance (compliance): 



E(/ , I P, = l)- E(y^ , I P, = 1)=[E(/,)- E(y^ ,)J/Pr(P, = 1) 



E(y'',lP, = 1) = E(y\lP, = 1) - [E(y^ ,) - E( y^ ,)]/Pr(P, = 1) 

where t is time and [E(y^ \)- E(y^ \)]/Pr(P\ = 1) = for / = 1 and 

[E(y^ i)-E(y^ i)]/Pr(Pi = l) = for t = 2? The last expression tells us that the average of 

each outcome for controls, which is unobserved, can be computed from known quantities. 



Model Specification for Looking at Cohort Specific Impacts and Between Year Impacts on 
Reading and Math Achievement 

Our analyses examined three hypotheses: 

• Average impacts on student and family outcomes were the same in year one and year 
two; 

• Cohort-specific impacts on students’ reading and math achievement test scores were 
similar within year 1 and within year 2; 

• Cohort-specific impacts on students’ reading and math achievement test scores were 
similar across years. 



To test these hypotheses we constructed functions of the impact estimates and computed the 
standard errors of these functions using the bootstrap method. The specific functions are listed in 
Tables Cl and C2. 

We used the bootstrap to compute direct estimates of the standard errors for several reasons. 
First, some analyses involve using more than one child from each family, which produces 



^ This expression for program impacts draws on Bloom’s earlier work (1984). 



clustering in the sample. To adjust for the clustering, we sampled families instead of children 
when constructing the bootstrap samples. Second, the estimation of the private school impacts 
involved the use of the IV estimator and is complicated by the implicit presence of interaction 
terms in the model when comparing across time impacts or between cohort impact estimates. To 
make these comparisons, we computed the functions in Tables Cl and C2 for each bootstrap 
sample and then computed the standard errors of the functions after 1 ,000 samples were formed 
and the models and functions were estimated. 

To statistically test the hypotheses that involved making multiple comparisons, we used the 
Bonferroni procedure. The Bonferonni allows us to control the probability of making a type 1 
error when making multiple comparisons. To use the Bonferonni, we can take the probability of 
making a type 1 error for a z-test, for example, and divide it by the number of comparisons 
made. For example, if the probability is . 1 0 and we are making 4 comparisons, then the critical 
value used for each comparison should be the value associated with a type 1 error of .025. By 
dividing by the number of planned comparisons, we implicitly set the probability of making one 
or more type 1 errors among the planned comparisons in this set to 0.10. This should be about 
equivalent to using an F-test in the usual setting when we want to test for differences among 2 or 
more means. 



TABLE Cl 

FUNCTIONS OF IMPACT ESTIMATES FOR COHORT 
SPECIFIC IMPACTS WITHIN YEARS" 



Parameters of Interest 

^1 ~ h \ ~^\2 

A 2 =/,| — /|3 

^3 =Al -^4 

^4 ~^ 2 \ ~^22 

Aj =Ii\ 

^6 =^2l -I 24 



Interpretation 

Difference in impacts for cohorts 1 and 2 — year 1 

Difference in impacts for cohorts 1 and 3 — year 1 
Difference in impacts for cohorts 1 and A — year 1 
Difference in impacts for cohorts 1 and 2 — year 2 
Difference in impacts for cohorts 1 and 3 — year 2 
Difference in impacts for cohorts 1 and A — year 2 



"Cohort specific impacts only refer to analyses of achievement test score impacts and not to 
analyses of family and student survey data. 
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To assess whether overall impacts and cohort specific impacts changed between years, we 
computed some additional estimates as indicated in Table C2. 



TABLE C2 

FUNCTIONS OF IMPACT ESTIMATES FOR BETWEEN YEAR COMPARISONS 



Parameters of Interest 



^1 ^21 
~^22 
^3 ~^I3 ~^2i 
A4 =/|4 ~^24 
^5 ~^\Overall ~ ^ lOverall 



Interpretation 

Difference in impacts for year I and 2 for cohort I 

Difference in impacts for year I and 2 for cohort 2 

Difference in impacts for year I and 2 for cohort 3 

Difference in impacts for year I and 2 for cohort 4 

Difference in impacts for year I and 2 overall 



The standard errors of these differences were computed using the bootstrap method and we 
tested for overall differences using theBonferonni procedure. 
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Appendix D 

Supplemental Tables Comparing Year 1 
Findings to Year 2 Findings^ 



^Appendix D explains how the between year comparisons were tested. Tables are numbered 
to correspond to the tables presenting the second year results in the body of the report. For 
example, Table 5 in the report has a supplemental table in this appendix titled Table S5. 
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